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FOREWORD 



; - This^^ is //the f irst-m series of :^ - 

General Information -Ofearbooks to be : updated and published 
annually by the National Assessment of 

These^ yearbooks are des igned to aid; the readers of National * 1 
Assessment's reports to understand the philosphies and goals : 
of .the National Assessment project, its methods for developing 
objectives and exercises, and its techniques for collecting, 
processing, and describing data. While these yearbooks will 
contain much general philosophical information, about National 
Assessment, each. volume will be oriented toward the subject 
areas, procedures, and techniques relevant to the assessment 
year currently being reported. This volume, therefore, is 
oriented to the year 02 assessment of Reading and Literature 
(assessed 1970~71) . \ The General Information Yearbook was 
designed to complement the modified reporting format beginning 
with the year 02 reports. . ' ^ 



The Reporting Format 

The results for the year 01 assessment (Science, Citizen- 
, and Writing), have been reported in what might be called 
a phase" format. Each phase volume treated a different aspect 
of the results for all the exercises within a given subject 
area. For example, .within the subject area Science, the ^ 
phase I report gave the national results for all the Science 
exercises; the phase IIA report presented results for the . 
regions of the country, the sexes, and sizes of community; 
and the phase IIB report presented results for races, levels 
of parental education, and sizes-and-types of community. 1 A 
major disadvantage of this format' is that it is difficult for 
our readers to follow the results . through the various phases. 

.. Because we feel that National Assessment results should be 
easily understood and both meaningful and useful to educators 
-fand other concerned persons, we have adopted an alternative 
reporting format with the following qualities: (1) the data 
are presented in a larger numbe r of re latively small volumes 
so that the reader will not be overwhelmed by the' sheer size 
of a report ; • (2) each volume contains exercises which 



•'•These 



groups are defined in chapter 3. 



together a way: ^^^^^^ is meaningful to educatdrs and scholars 
iir^he relevant subject area; and (3) along with each exercise 
the reports gi^ alT data2--national and^^^^ 

that exercisie. K ; ^ : ^:^^:_vf^ ' _ / ^ : : ^^v— ; ^ 

^^^^We^ believe meaningful 
ta ^ucators^^ ^e pal 1 themes y A theme is i 

a :Se$:^;df exer^^^ share avcomnion Tdeav b 

require diverse b^e^^^ responses from resppnd^^ 

eachrsubject ^e^v-^ 

summary volume w^ ^ 
relevant t^^^ the subj ect area (includihg^^_t^^ themes and objec- ' 
t i ves: for the area)^ and ±he - general trends that: appearVin the 
dataV; A separate volumei^is de^^ which the 

data ; for the -specif icve^ the theme are given 

aicmg-wi^^ja. summary^ v 

Released vahdvUnr^lea^ed^^xercises^ ^ - V y \ 

^ > "Not all the^ are selected to be 

::^a^ sed^^^^^po^^ 
^fe^^s^^aiSies;^^^ 

imr^Maj^M^'^2^ in 

^^]hF^)e^^c6ii§^^ 
^^b3^ipS^ 

^Shapte^£S?^^^ _- 
^ei 



t 

t - 



TaiBj rel^sed^^ex^rci^^ ^eetv^s^i^fiedL a^^i^^di^ 

^ number. The first digit identifies the theme 

7>?ffi^Sthej^^^ 

^entrfy^ i^ie^ ^^ithiiS the ^S^^ 

fqjarthj^^ 
^jiere^apj^^^ 



^^kqepl^i^p^q^T^^ ^ reielSch^ 

fvolumes • V V : v j^^ ;^ V v — - y^^^^ 



M3 giyes^the 
^p^f ^themes jvd:th^ 
^)f iwha t : tlieme sSaire 



ration aleripf^ theme s;^ 

ect area / J and^fan expanded i def^^^ 
thei=r? function i ^.^^^ 



-3. 



ff- 



-= - vi 



- < 

-1 



.-if 



'-- Volume ; Mumbering System ^ 

system_ IS being inaugurated : whi ch - reveals the as ses sment 511 
subject area; and: vpMme; content. A: volume number, tSrefo>e^' 

referring to the assessment, year, for example, 02 for the 

"^^^ an initial ietter 

or letters for the subject area as follows: J-etcer 



A : 
COD 

C. 

L = 

MA 

MU 

R 

S 

SS 
W 



= Art 

= Career and Occupational Development 

- Citizenship : : / : / 

= Literature r ; - ^ : 

= Mathematics - : " 

= Music - 

= Reading ■ : 

= Science " r . 

= Social Studies ^ ^ - : y ' 

= Writing ^ : / ■ 



The third part is a two-digit number indicating both the • 
content of the volume (summary,' theme, or research) and the 
order of publication as follows : = - ' 



00 
01 
19 
20 
29 



^^^aumma^yif^^^ 
^j=>obs§rvqd 

€: i;^-VS V- ;^^t hrpu^hW Sl^-Vv ^^^^ - i 
= observed results for thetae 19 

= numbers for research volumes such as 

reports of balanced results and 
z others i^s^f^ ^ ^^^y^ ^ ^ 



For example, the 
is 0 2 -R- 0 5, and 
theme 2 1 s 03-MU-O 2 . 



Summary is 02-R-OO, Reading theme 5 
balanced results" is 02-R-20. Music 



V : . • Limitations of the Data 

The National Assessment of Educational Progress was ■ 

ce?nel w^?h^;S scholars, and lay persons con- 

cerned with education with data regarding the educational 
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achievements of various groups^ of young Americans in 
10 :subject areas.5 Within the limits ot errrbr dueVtb m^^ 
ment and sampling error ^6 the obtained, data as^^^ p 
National Assessment reports accurately describe the educational 
achievements of these: groups as they act^ually exist in the real 
world. These obtained data portray? the reMi- problems facing 
educations-improving the educatid^^ of various 

groups of stujdents* - ^ ^ / - : ^7 ^ 

7 When the data show that a grpup_ has \achieyed eit^^^ 
brl below the ^nation as a wholes one : mu s t exe r ci s e extreme 
caution in attributing causation Ttothes^ . 
Igatipnal Assessment is not intended, to provide reasons -fe^^^ 
dif ferences ; its purpose is to describe such Vdif fer^^ 
they . exist . Many: factors may affect a 

to - give acceptable responses to -exercises^ in the = assessed 
sub j ect areas . Cdnsider . f or Je a hypothetical :^group 

whose achievement is well above ;^the7ha Most ' 

members of tne group : may att^hci^ 
phygicatt faci^M^^ t^^nd^^h^^ 

h^gfi^ sjbbipr^cor^^ ~_ 
:s6vpn]^ Jth^sey fac^ 

The Or^aITfe^ of : av^ipupSi^ ^ 
w^J^]^ a t :tHe a^tbr^^;<ga^;{<^^ 

^ayi ahi^ dif f eii^nc^ :^iit^ ~~- 

?iiftibri Jas : 

gt&up . In other :^pf^V^^^^^ v^;v 

necessarily beirig 7th^ cause3p^^ \ : V? 
differential achievement between that group and the nation as a 
whoie.^^ - r 7; : ^^^T^^sTrS^fiV^^ 



I - 



I- 



:??-5: ofterr. member ^ 

d^kcr ibed abbve:,^re di gtgttojit^[C^^ ^^a^^^^^pj^PE^ 
ti/Snately largg perx^^^ 
g^:oups ^hich cohtribut^^35o?t^ 



"^The^ groups ai^e7def4n:eJ4 i^ 

footnote 1> chapt^ ;t. v ^ : v : 



^^ S e e c h a 11. 
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achieyemen^. The data/pbtained from these groups do not - ^ 
allow one to evaiiaate; the effectiv^^^^^^ ?he Educational > 

SrJni- fetors. A statistical procedurl called 

balancing ad3usts for thie disproportionate d of 
; group_^members in other categories.:, or groups for which there 

^entllSlor'S: W^'' Thi . pr?cedEre givef ?he'acSeve^ 

4.5^ ^""F^P "-"^ would have been 

tlSn^Kl f the rnembers of the group been distributed propor- 
AssessmeL hS?" - W ^^^^5°"es or groups/ NatiSSr ^ 

a?f nreSetpd fn ^^J^"^?'^, ^ °^ dxsproportibnate representation, - 
are presented^ in a special research volume . Agai^ extreme 
caution must be exercised; in interpretiiig the balan bed lata 
The balanced^data sti II: reflect many extraneous factors no? ' 

m'^.SSi'c^''^^ therefore are's"m°^t : 

pure measures of the achievements of the group in question 

Se JiSl^Sf f H 1.1 constru^ as 
necessarily^ bein^^^ the.cause or even as beirig^^ a cause for the ' 
|if ferento^ achiey^nt 



- Exhibit F-1 gives the number of 
^xercise -pacJeag^ 
^ad. Mne ^patate^^ 
Oth^ JiiieO^^din 

ire unique at each 
^s^ not the same ait 



to whom each 

_ ^ach^ag^:^ level. X iiorizon^^ 

group-administered packages (above 
the individual-administered packages . Packages . 
age level, therefore, package 5 (for example) 
ager^vCls/^g^ / - ^ c-^ 



mm 



Pkg. No. 

- 1 • 

2 
3 
4 

r 5 

6 

7 

8 

9 
10 
11 
12 
13 
14 
15 



2610 

2518. 

2595 

2589 

2625 

2595 

2558 

2548 

2573 

2201 

2210 

2195 



Exhibit F-1 

, 13 

2602 
■ 2543 
2550 
. 2559 
2537 
2550 
2509 
2542 
2540 
2583 
7 ^ 2534 
. 2591 
2547 
2197 
2199 





Adults 


2405 


1241 


2414 


1259 


2361 


1251 


2428 


r 1254 


2332 


1242 


2308 


1226 


2394 




2354 




2343 




2322 ■ 




2125 




2145 
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GENERAL INTRODUCTION 
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History and Purpose of National Assessment 

By the early 1960's many billions of ^ ^ars were being 
invested annually in this formal education of our young people. 
The only ayaxlable measures of educational quality resulting 
from this investing bad been based upon inputs into the 
educational system^ such' as teacher-^student ratio s^ number of 
classrooms^ and niu^ of dollars spent per student The 
tenuous assumption had the quality of educational 

outcomes — what students actually learn-rwas directly related to 
the quality of vthe inputs into the educational s _ 
significant direct assessment of educational outcomes had been - 
made/ The typical state--administered or^^^ scho^^^ 
; achieyemept tiests,. which provided scores whereby: one stud 
^qould^e^jconpa^ u^ef ul:?if br^jc^tecfb 

siudents ; b providedV very^^^ littlerinf ormationvabdut what 

Vstu4entsvvwere^;actuall:y^ 

^ ; V: Thisi^insuf f 1^ cfoncerii of : ^- 

^FraricisVK^^^^ of v Educatio^^^^ / l 

( 1962-l^e5).^^^ w to." f ind wayS^ 

^15r^hi<?hllt^ of^-theses^^^ 
icbjrference^ John WV Gardner^; piresrident ^^g^^^ 

?2tlon ^ asked a distinguished and^ lay: per sons 

^to fprrci thV^xploratoiy Progress bf^: 

\Edvication \(ECAP^ >^ This^^p^^ ^ 
:5Pyler^ was: ta ^xagiine t^^ assess- 
f^rit^of educational^ a^ \-- 

. After much study, ECAPE deemed that it was feasible to 
inaugurate an asses smeht fill the /inform^^ 

r^g^rdiag . the quality-:^of educatibrval o^^ 

assessing the knowledges / :undetstandih^ attitudes 
^In 10 subject areas! at four age levels (9,13,17, and adult — 

ages 26-35) . The project began its first assessment of the ^ 
:^suhj ect areas Science > Cltiz^^ an^ Writln^^ , 



^Artf Career and Occupational Development, Citizenship, Literal 
ture. Mathematics^ Mu^^ Social Studies, 

and Writing. : . : \ : \ - - - 
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of 1969. Later that*" same year, ,the project came under the 
auspices of the Education Commission of the States; and was 
named : F * tional Assessment of Educational Progress *(NAB-P) • — 

>?or tne first time , there would be ' a: Cdireo t measure of 
educational outcomes which could be utiriz*=^d by school systems 
to improve the educational process.- Since NAEP is to be an 
ongoing project, it will eventually be -able to as^^sess; chakges 
in these knowledges, understa^^ skills , and attitudes to 

determine any changes in educational outcomes .2 

The Philosophy of Assessment ^ 7 / " v V ^ 

- The typical achievement- test measures people . : rndiyiduals 
respond to a number of questions or tasks ( i tems ) , and a score 
is : determined for each individual . - On;;.the basis ^f ^any s uch 
individual scores , onez individual can: be compare^^^^^^^ any other 
or with the mean (Average) score for: the entire gr^^ Little 
attention is paid tovUie specif ic knowledges or skills possessed 
either by the: individua^^ by ithe lentire g : 

;^^§jaM on VA^ 
skiSl ii^ ^an^ :: att^tt^ 

^ath^> ; weVjo^ -\ 
age^ev^ls niehtijoiiedV^^ 
gi:pu^3^73^9 !f re zJSd 
i^efl^qfe^spqci^ig^ 

;KtM^ltud§s^ " 

V:4r^EiJ^^Qnal^/A - 
guestixyis^ or t^sks^ muph7 the:;:s^^ --_ , 

acKiervem^nt_ te^ Slnde we d6>^iot Tohtain^ir^ 



^tKIs 3e<3tion: was^ 
Drg^^r ank^WomgXc ^hd^ ^ s e sumen &^pp ro ach^o ^ 

Exercise 13e^^^bpmeht by; S. _ 
B0±d_ie _an;d^-may^^b^^bt^^ 
Educai^^ 

Linchpin Tower , £$ eiO^ilncplhv &203^ 

•^Naisional Assessme^nt^ s groups^ a^ ^ ' 

cfeptjer 3 7 ^ ; ^ '7^^ ^ 77:;: y ^^ > _ v-^:^ ^ 

^Chapter T discusses the ways in which 7N data 
are described. ^ ^ 7;- -^^^^ ^^:^^^^^r: - 
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these sets of questions or. tasks ^ we call the sets "oackages" 
rather than tests . We- also eaid the -questions or tasks 
exercises" since they; allow the; respon^^^ to demonstrate 
whether or not he possesses the knowleage, understanding, skill, 
or attitude to resppnd acceptably^^^^t^^ - _ 

^ All standardized achievement test i^ems ideally are of 
medium difficulty to- assure maximat^^^ Assessment 
^exercises are equaMy represented by: easy / medium , and hard 
: difficulty levels ,; i.e., we expect certain knowledges , under- 
standings, skills and attitudes to be possessed by a larger 
percentage of individuals than others. ' > 



- _> 
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Once these jediu3ay.c^^ Identif ied>/ eier^ises 

yoimg^ pebple ^reistchleyihg^^ 

berately used to ^^is from 
st<aiidard^^ 

tl^fevi^r^^j^^ 

some gxou^ q£ peopip^^r^^ 
i^v^o^measurg t 
atti^des of. laf^ 

of v^a^t^^a in^ . Theses uOrts 6f^ as s jes sme^it are^ reported in /terMs 



CHAPTER 2 



DEVELOPING OBJECTIVES AND EXERCISES 



Criteria for Developing Objectives and Exercises 

One o f the mo s t impor tan t f e a ture s of N a t iona 1 Ass e s smen t • s 
developmental work is the formulation of educational o 
which govern the direction of the assessment in any given sub- 
ject area • These ob j ectives define a se t df goals which/ are 
agreed upon by both laymen and educators as desirabre directions 
in the educat ion o f young peppie . For Nat iorial : As se s smen t , the s e 
educational objectives must be acceptable to three ma j or groups ^ 
of people. First , the objectives^must be c^ 

by scholar s in the discipline of a given subject area. Second, 
objectives should be accepta^ to most leducators dnd be con- 
sidered important t^ goals in most schools. . Finally, and 
most uniquely / lay- citizens: interested in education must agree 
that the objectives^ ^a^ important for yoAirig p<eople to attain and 
Jihsa^hesej :dbge<^^ >\ ^ ^: 



r 

i- 

r 



^1 

1 



_ j^rbin j^aeh^^^l^sj^^ 

f ey^nrt- gebgxra^ s ec tjjohs ^jcj^ 

vajeWp<y^l^-/^^^b 

ati^cti Jlss ^sSnienfe ^ / 

IpttioMir^ 

educators and lay people agree, but rather that the final set of 

ptg^gfives for:^^ a. 

cmgsg/secrtx^ 

zens^3^n; th^ 

li^ weilj^as^^i : 
:&^the5^ 
qJDj^ctiyes ctorxeht^^ 

i^fpnnat 3^ tb^ ^the^ ev aluati qn^pif^Am^r^^^ 

t hes^^ bb j ecSp^ 
meht Vcyele: 
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of individual exercises; for example, "90% of all 17-.year-olds 
know the name of their states, but only 20% know the name of < 
their Representative in Congress . " - In addition to providinq 
information on the achievement of ^-,1:3-, 17-year-olds and 
Adultsy ages 26 to 35 ,: for the nation as a whole , National 
Assessment also provides results by. regions of the country , 
siz^ and type of community, sex, parental education , and color. 1 

Because of the uniqueness of the National Assessment 
approach to information gatherihg^the following critoria have 

^ = Content Validity. Every exercise must be a direct measure 
Of some important • knowledge , understanding, skill, or attitude 
that reflects one or more objectives in a subject area. An 
exercise must be meaningful , and directly related to the obiec- 
tive that It IS intended to fulfill. In practice, then, an 
exercise has content validity if it makes sense to a reader 
who sees- it together with an objective and says, "Yes, this is 
3 good measure of the knowledge or skill called for by the - 
objective." The chief criterion for determining content 
validity is that an exercise pass extensive reviews by subject 
matter and lay groups (described later under procedures for 
exercise development). But to further insure that its ' exercises 

meaningfuland relevant. ^Jational Assessment involves 
young people at the ages being assessed in both the ' 
and review of its exercises. 



Clarity . Every exercise must be easy to understand so that 

^^^^ ^® is being asked to do. This means 
that the directions and formats which go with the exercises must 
be simple enougn for anyone to understand and the vocabulary, 
iraseology, and length of sentences must not be confusing. 

. ' Ty Pe^:o^ Exercises and Formats. National Assecc^n^on^- - " ^ 
instructs developers of its exercises to use that format which 
provides the best and most direct measure of the objective 
being assessed, and to create exercises which obtain actual 
samples of a young person's skills. In doing this, they are 
also encouraged to use, whenever possible, individual inter- ^ 
views, group discussions, or observations of group tasks to 
supplement the usual paper and pencil exercises. NAEP asks its 



----- . : ----- . 

Groups: within • these categories are defined in chapter 6 
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exercise writers to experiment with unconvieritional stimulus 
materials such as pictures / tapes, films or practical everyday 
items in order, to heighten interest and bring greater variety 
to the format. Most importantly, NAEP emphasizes the necessity 
for developing a pool pf exercises which will reflect the 
cultural pluralism that exists in America and which will have 
meaning for all groups in our society, / _ 

: ^ Difficulty Level . Since National Assessment does not intend 
to: rank people in order , it is not appropriate that its exer- 
cises be of medium di f f icul ty as usually : they are in standard- 
ized tests. The intent of National Assessment is instead to be 
able to describe the Tcnowledgbs , understandings , skills /and " 
attitudes of the - mo s t and- leas t abtle rypung people as well as 
those of the average young pier son. / Therefore /'it develops 
easy , medium and: difficult exercises which are aimed at each of ^ 
these three groupsV r ^ - - v ^ " 

Overlap Between Ages ; In NationaljAssessment an exercise 
whicli "overlaps" is o:ne that is appropriate ^fp^^ than one 

age level . i^^^^^ A^^ 

^aPg^er4ley^^ :m^y?^werl^ 

jen^ii^ ageS^l?^ \ si^ 



f ^hstructlbhs foraScbrxhi^^ ^^ jgg:r^ 
^eiided'^jbypes^t^ 
^^§g(bSt^dv^ Be^ 
^xerM^s^^^r^ttefcs^^^ 
^^cx^re ;an^ 

t;iei^edt,on£^:;^ basis/ jto^Jin :^c±uai^ample^ ^he\ 

-r^eBul ti?:Ja^i^ vz.^ 
^the^ :^epring^^^^ ireppirtirig^^ i^^^^^iT^^ 3 ; , 
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1^ 



p. 

T 



]^~[^TQ^6^ f or^E)e\;^bpirfg^ 

^^1/ ^loising the f i^st iyear^;^of^v^ 
merit SojE^ oh^ gptiy^ :jand,^^xe rci^se s^ \^ 
i^ge /^JOJlti^ 
^grvrce:/^ Sqiehcer >Reisearch^^spbr^ 

^&r. Reseafeh^ Nationai^^s^e^Sfe^ : 

smaO/l ,;vand N a ti pria ]/ ^As s e s sifent'^: role ^ip^ exer^o^ ;dev€^ lop- 
ment was to act as a guide rather than a direct supervisor. 
However, since ^970 / therb h<^' beeiv a Icon^^ 



'ehapter ,6 gives a detailed- account of the scprihg; procedures, 
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the, size of the National Assessment staff, and newly created 
interdepartmental teams have brought about a greater 
coordination of developmental activities. In some areas/ sub- 
ject matter specialists have assisted the. Assessment in the 
preparation of objectives and exercises.. The developmental 
process currently in use incorporates previous developmental 
experiences , and provides for closer monitoring by: National 
Assessment staff of its contractors (individual specialists; 
as well as large testing orgranizatibns) . : Thiis development 
process is .divided into five major phases: 



Development (or revision) and Review^ of 
Objectives and Prototype Exetcises ^ 

Preparation of iSxercises and Exercise 
Development Tryouts. 



Phase A 
Phase" B 

Phase G - Review and Revisions of Exercises 



Phase D JB\ield-;^^ 

y__ — ^-^^V^Resul^a 



ting, ^nd^ Re yijew p f ,Tr^p_u t 



^^3^1^ihg^ 

^gh^as^Va^-TV i^^ 7; 7^ 

v-^ ^f^ C^b^^cfeiyes /ai^ Wi^ 

,. This phase allows for reconsideration and possible revision 
of objectives in a previously assessed area for the develop- ' 
ment of objectives in, a new area. A variety of inputs are 
sought in this early phase. The developer of material reviews 
ttie literature in the field and collects appropriate informa- 
tion from universities, state departments of education, school 
districts, and objectives exchanges cfround the country in 
order to determine the latest ideas in objectives develc 
Subject matter and objectives specialists formulate new 
objectives or revise the existing ones. These specialists, 
representing- a broad cross section from around the country ' 
are selected from elementary and secondary schools (public' 
private, and parochial) as well as from colleges, universities 
and other professional organizations. All of these various 
inputs lead to the drafting of a. set of objectives. ----- - 

, The next step is^ to hold a review by a separate group of 
;su6 j ect matter specialists ±o critique the new or reformulated 
QDjectives. Again, as in the case of the specialists who 
helped formulate the objectives, there is broad professional 
representation. The specialists evaluate the objectives for 



1 



content validity, appropriateness , relevance , and reportability. 
If three reviewers think it necessary, the objectives may be 
further revised 'and refijied_* ^ : 

\ In the process described above, both subject matter 
specialists and educators participate. The next step is to 
seek the reactions of lay citizens. Lay people from. ^around the 
country are invited tjo attend al review of -the object iv^^ at 
which numerous panels are convened with each panel having 
representation from different occupations, socioreconomic back- 
grounds, and areas of the country. Coirimehts from these panels 
are used as the basis for further refinement of the objectives. 

VJhile the objectives approach their final form, exercise 
writers develop prototype exercises. A prdtptype. exercise is 
one which serves as" a model for; the btaer ^ 

ejxercises; it must be a measure of an: ob 

clearly stated and include a rationale , directions to adminis- . 
trators , scoring ins trubt ionS:, a key or; samplev of acceptable 
and unaeceptab]^^ - : 

^r^sUits> j^^Ilh^ rd^as^^ 

^fa^^ect mlitl^er sp^cora^ 

withih: jtSie; f ieMClcnd :pec)^3S^ enoMT^^^ ieyelis J) f 

;:g^£f&inn[iahcer^ are^piAced^^lV^^ ^ v# 

^kerca^;^g^ 

^f^om trie^ t rjioi t s r^m(J h§ pr c^tt)^ t]*ero|f^^ theix: 

i^kiew^dtJ^i^ .an^ 
>d;eMdSL§: :edd^ 

^^iiDther^ ^ecpnuneria^ t^ Refined x^r tha^ 

^^fie pri^djaci^ioivJ^^ 

r c£ses^r,gu idih^ ^e :cbhtra^ vtyp^^ crf^-ex^^ that ; 

snbuidL be 



^hase \ ^!^epcLraLl^ x ^ r^^ ;: v?: J; 

?: jg^? ai:xe^^ ry bu t s/ ^^l V ^ 1:^> - : ~- .-^^^^^ s^s :-\ 

The numbWr^b^ the nunfcer of 

i>|Lirticutat^r:^ STirr\ui^;:thafc>^ 
^xlst ; (•unreieau5ed^:^xetcises^f^^ :; ^ __ : 

exercises -an : a pre^ . 
as^ vciried \a:^ange^^ of the ir 

writers who: he Ip^ Once he 

has , a pboi of :new oexercises^^^^^ cbritractpx cgnd^^^^ limited . 

^tr^but to^obtaih=:n5^me sa^ factual i^esponses. Ideally, this!: 

:^^oat shbuld^be sor that they; ; 

can obtain some datS b^^^ of the task, scoring cate-- 

^ries, and axiministrative^^^ -Another purpose iDf the 

;trybut is to provide ii^^^ 

V^j^viewers about the results the exercises will yield and the^ 
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possible problems that will have to be investigated before the 
actual assessment, 

Phase C: Review and Revisions of Exercises 

l-Iational Assessment submits the exercises to both subject 
matter _ specialists and lay people for their review and 
criticism. A panel of such reviewers is formed with the 
following considerations in mind: 

There should be ■ ' " 

1. Representation from ail; parts of the countr^^^^^ 

-2. Representation from, different specia^^^^^^^^ 

the discipline (academieiahs and other specia^^^^^^^ 
: working^ in professional- capacity es i 

3. Representation /from organizations or projects 

; ; : jconc^rTii&d;^^^^ rarea/ v. : : ;^ \ - i/ ;, v 

4. Public, private and parophial school people ^ 

" (classroom teachers, curriculum specialists, - " - 
: administrators) - both elementary and secondary 

r^5>^;y ^ - 4evers^i> ^ -"^^^-5 - ? - " V - v:> ^ ^ ^ ^ ^ 



5i: 

6. 



Persons knowledgeable about minority groups 

Persons knowledgeable about low socio-economic 

pi^blemsr : V: "^v ^ , ) : ] : - \ v , 



8. A balance between male and female participants 
For the selection of lay reviewers, the following criteria 



•••• There should be geographical representat on and 
^ people from various types of communities, i.e. , 
towns, rural areas, big cities, etc. >: • 

2.,- Representation from different national 
organizations 

3. Representation from different occupations 

4. Persons knowledgeable about minority groups 
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5. Persons knowledgeable about Ibv/ socio-economic 
problems . _ 

6. A balance between male and female participants 

The subject matter specialists in their review of the exercises 
are primarily concerned with content validity ^ relevance / 
administrative feasibility / completeness of scoring and report- 
ing instructions; the lay reviewers are primarily concerned with 
^appropriateness and relevance. 

Phase D; Field Testing and Revision of Exercises 

Af ter the exercise pool - has_ been revised according^^ 
suggestions made by the reviewers there is a full scale iryout 
involving representatives of the community / socio-ecpnomicv 
racial, sex / and;^ regional groups that are f ound inr the - 
actual assessment. Thevpurpose of the tryout is _to_ give some 
indication of how-^each exercise will -probably function in an 
assessment /^^^^^^^ ^^hat can^^^^b^^^ to ^ _ ^ 

iiimEyrcwe-ato :^ V 



.FpXIowiJi^Cthe.: ibryonat^ subgeit ^ thet^ 
tilts - to>:^m4kep^Aire^ ^:h^r=t h^;sjcbi^^ ^uide s^ fef lejc t^ actual v; ^ 
vire s pons e s and: will" prby ide : iitfpo^^ de s i^'aB 1 e i n f orma tionyi ^ 

^Ph-as e E ^ FoJira l^^:Re vie wis? ^ v^:^^ ^^^^ v"" - " ^ ^ 

^ - At thls^ pc)int; xa]^ 
reviews -byi bo^^^ matter^^^ s^^ 

a[re cahdi^atie^ Hpwevj^^ nu^ -i/ ;^^ 

jexerjcises :develop^^^^^ 

vAised^ in\ja^^ _eacti " 

::exercise stc«cb2rdi : ; ;v 

^iatingfs are: thhh jrire viewed :by: ascommi^^^ 

matter a measurement^spe 

cpntractpr staff member^^^ 

invent of thC^^ea=; desi^a^es rthe;^ ex^^ 

>toe admi^is tered::in j^th^:^^^ oi: the 

^xjuality, ratings: and reporti^ needs^ pbj^ctiyesl> 

The selected exerc is^e s are the^^^^ f bjr^ard^^^ to the Unit ed States 
^Office of Educatiori to be checked for: ah^^ / : 

privacy on the part of respondents or. possible of fensiveness . 

Additional' Development Work ; 

During preparation for the^ packaging of exercises and 
printing of assessment booklets {packages) in any of the areaS/ 
exercises aie closely examined by members of both the National 




I 



Assessment Team as well as contractors responsible for the 
packaging, printing, and field administration functions, in 
some, cases / administrative instructions are added to facilitate 
administration in the. field. 



. , ' Development of Objectives and Exercises 

Year 02 - Ileading and Literature 

The development of objectives for Reading Was the contract 
responsibility of Science Research Associates, and for Litera- 
ture this responsibility was handled by Educational Testing 
Service. In both areas this was the f irst ^development of 
objective's,, and this development followed the process described 
under Phase A of the previous sectioh on Procedures for • 
Developing Ob j ectives arid Exercises . j ' — — ~- 

V The development of exercises in Reading was iriitially 
handled by Science Research As sociates,. However, following the 
siabject^ matter review of the Readijrig exercises -in Phase C it 
w^s^epideds; tha^^ ^tcx ^be 

supplemented with additional exercises. Independent writers 
were contracted to develop. these additional . exercises which 
were in turn reviewed by subject matter specialists and lay 
citizens and then added to the existing pool of exercises. : in 
Literature the development of exercises was completed by Educa- 
tional Testing Service. Except for Phase B exercise development 
tryouts, procedures for the development of both Reading and - 
Literature exercises followed the process described in the 
: section entitled; Pfocedur es for Developing Objective s and 
Exercises . The Phase B exercise development tryouts were added 
as a stage in the developmental process following year 03 work 
to permit the detection of deficiencies in exercises early in 
the developmental process. ^ 
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CHAPTER 3 



DEFINITION OF NATIONAL ASSESSMENT GROUPS 



We report results for four age levels which represent the 
completion or near completion of significant educational levels 
by most individuals at each respective a'gerl 9-year-olds, most of 
whom have completed the primary grades (kindergarten 
grade 3) ; 13-year-olds, most: of whom are nearing the completion 
of the intermediate grades (grade 4 through grade 8) j 17-year- 
olds ^ most of whom are nearing thevcompletion of /the secondary 
grades; (grade 9 through grade 12) ; and Adults aged 26-35, most 
of whom have completed ali formal "education. 2^ L each of 

these. age levels , we report results3 for the nation as a whole 
and for geographic regions of the country, colors ^ sizes- : 
types of community, and; levels of parental education as follows: 



Character is ti c 



^ex" 



Region- £ p £ ;^th^ ^ 
Gouivtry' ^ 



Group 
Male 

vjpeml^ ; - ~ 

Northeast: 
^oytheast 

^Cent^^a^-^vz 
West 



Color 



Black 
White 
Other 



■^Some indiyidjaal^ ^ 

same edUcationai^^^s ii [ s^ 

^The - actual grades in^^ the ^primary , intermeciiate-, and 

si^condary classifications may vary a^ systems. 

^Chapter 7 explains how National Assessment results are 
described. _ 
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Level of parental 
education 



Characteristic Group 

Size-and-Type . Extreme Rural 

of community Extreme Inner City : 

Extreme Affluent Suburb 
• . Inner City Fringe 

Suburban Fringe 
Medium City 
■ Small City / 

No High School 
Some High School : 
Graduated High School 
, Post High School ■ 

For the year 02 assessmentv these age; levels and groups are 
defined as follows: , 

Age Level - - 

^, ■ 9-year-olds: all individuals born from January 1, 1961. 
through December 31, 1961. ' ' 

13-year-old s: all. individuals born from January 1 . 19 57. 
through December 31, 1957... ^ ' . 

17-year-olds "in school": all individuals enrolled in 
secondary school and born from October !, 1953, through 
September 30, 1954. i-ni-uuyn 

. 17-year-olds "out-of-school" : all individuals not enrolled 
in secondary school during. March, 1970, and born from October 1, 
^•552 , through September 30, 1953; and all individuals not ^ 

o!;f°ii^'^/''i 0??°''?;''^ ^"^i^g January, 1971, and born from 

October 1, 1953, through September 30, 1954. ■ 

March^3lt^l945!^^ ^^^^^i^^^ls born from April 1, 1935, through 



Region of the Country ' : ^ : - 

^ National Assessment uses the same regional division that is 
used by the Office of Business Economics, Department of 
Commerce. ■ . 
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Northeast 


Southeast 


Central 


West 


Connecticut 


Alabama^ 


Illinois 


Alaska' 


Delaware 


Arkansas 


Indiana 


Arizona 


District of Columbia 


Florida 


Io>wa 


California 


Maine- 


Georgia 


Kansas 


Colorado 


Maryland 


Kentucky 


Michigan 


Hawaii 


Massachusetts ^ 


Louisiana 


Minnesota 


Idaho 


New Hampshire 


Mississippi 


Missouri . 


Montana 


New Jersey 


North Carolina 


Nebraska 


Nevada 


New York 


South Carolina 


North Dakota 


New Mexico 


Penhsylyariia 


Tennessee 


Ohio 


Oklahoma 


Rhode rs land 


Virginia 


South Dakota 


Oregon 


Vermont 


West Virginia 


Wisconsin 


Texas 








Utah 








Washington 








Wyoming 



t - 
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Color 



^ individuals Blac^ ^WI^ ^ or J3ther on 
tH&^basuts of^^infbrmfa^ for 
Blac^s^:^(^^ 

as^K)ttie^^rw - _ 



^ S jbze ^d-NTy pe: j b^^ G omitm^ - ~ v ^ 

l^ T^ ^ize^hia^ype pfvcoiiumin^^ 
^hi^^^^xtiSer^^ composed: xTf ^ 

apE^rpximate]^ withi^P^i^ ^^zes of com- ^ 
muni^y r^p^resentirig 70%^ of the I^^^ . 4 

NQTEjr - ^fie^ifpur^ the 
STOC classif icati^^^ eqMvMeiat ^to the_ f Qui: srz^^ 

iconmiunity^ categorizes w 

Ir^tti^r do npt ^hayjB^^^^^^^ y 
:ext*acted f rom^H^them/ V^;^ > 



e year 02 ^ize;^of-coiranunity c uppn the 

ppp^ulat ion pf tSe cpiranujh^^ the sclipol being - 

assessed is located, ihe year Ol size-of-^^ 

were ba^ed upon standa metrrpppli^^^^^^ areas (SMSA) ^ 

4:heref ore every iridivid^ tn an SMSA defined as' a particular 
Sbc categpi^ wasrciassifie^ to that SOC. The four 

sizes of community are Big Cities |p Urban Fringes^ Medium-Size 
Cities> and Smailleir Places. 



i . - . -^^ 

-— - - : 
^ -- -1- - "^ 

ill 
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^ ^ -The definitions of the extreme types of community were 
"occupation question" for the school (see 
, Exhibit 6-1) . One extreme group was selected- in each of the " 
■ three directions indicated by an exploratory analysr.s: 
(1) ' schools in rural areas where a high proportion of . - 
the people were professionals or factory workers, (2) city 
schools where a high proportion of parents were either not 
regularly employed or were on welfare and. a low proportion were 
professional or managerial, and (3) near-city and city schools 
Where a high proportion of parents were- professional or ^ 
managerial and a low proportion were factory or farm workers, 
not regularly employed or on welfare. 



Exhibit 6-1 



In-School Occupation Categories 



Principal Vs questionnaire 
V categories^ 



or man 




Sales, clerical, technical , 
or skilled workers 

Factory^: or other blue collar 
workers 



B 



m 

m 



Farm workers 

Not reguiarly employed 

On welfare 



re; 



It 

I 

5 • 



year 02, there were no extreme type-of -community 
categories defined for individuals out of schoor since we did 
: not obtain occupation data from Adults and 17-year-oids not in 
school. 

..^ Smaller extreme TOC (^:ype of community) groups (less than 
:10%) would have been even more rspresentative of the respective 
extreme TOC's;, larger extreme TOC groups would have had more 
reliably determined percentagea of success. The sizes of the 
three extreme groups—approximately 10% of all individuals 
.assessed - were chosen as a compromise between more represent- 
ative extremeness and greater reliability. In the definitions 



I 
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below, the capital letters represent the codes for the categories 
in Exhibit 6-1, and the percentages- refer to the year 02 
assessment. 

The four size-of-community (SOC) categories [before the 
three extreme ty^^es of .community (TOC) have been extracted from 
them] are as defined as follows: 

SOC-1, Big City , This category comprises 21.16% of the 
total sample and represents those individuals attending schools 
or living in a community within the city limits of a city 
greater than 200,000 (compare to STOC-4). 

SOC- 2, Urban Fringe . This category comprises 22.68% of the 
total sample and represents those individuals attending schools 
or living in the metropolitan areas served .by a city with; a . 
population greater than 200,000 but outside the city limits 
(compare to, STOC-5) . ^ 

- _ S0C-3, Medium Size City . This category comprises 19.07% 

ofithe total sample and represents those. individuals attending 
schools or Tiving^^ 

and ;;2:00 ,0^00/ T(;comf)ai:e t^ ST0C-^6)^. i , — : ; " 

"" v SOC-4 , Smaller /Pracesv This category com^ 
ofsthe total sample and represents those^^ i^^ 
schools or living in a cbitimunity m population less tJaan 
25,000 (compare to STOC-7) . . 

The seven size-arid-type of community (STOC) categories 
are defined as follows: V 

STOC-1 Extreme Rural . This/ category comprises 9 .15% of 
the total sample and represents individuals: attending schools 
in a community having* a pppu^ They are 

among those ranked highest on the rural index (C+2A). The 
communities comprising the Extreme Rural category were located 
within the following three SOC^ categories: 



^The year 02 TOC categories were extracted from the year 01 
SOC categories. 
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Urban Fringe; " 0.18%6 —within SMSA? counties containing 

ity with a population greater than 200/000 but outside the 

V' limits . 



a city 
city limits. 



Medium Size City; 0.16%— within all other SMSA counties 
°?!! "'•"^.L^'-^^ with a populatiori greater than 200,000/ 
b:?we°enl5?SSo'and ^ city with a population 

not iSH^^^e^irsI^^^^ 

of i^i T^nllv l^l'^l'^^ ^T^"" ^^^y • '^^^^ category comprises 7.25% 
of the total sample and represents individuals attending schools 
ma community within the city limits or residential a?Ia 
served by a city with a populction greater than 150>000. They 
m r^ZtJi°^^ ranked, highest on the inner-city index E+F-a!^ ■ 
The communities comprising the Extreme Inner City category were 
located within the following three SOC categories: 

V Big City ; 6.45%— within the city limits of a city with a 
population greater than 200,000. ' ^ ^ 

y;^ ^ Urban_F^^:, ;o^ ;^ 
^ity li^tL^- — - -"^"^^^^^-^^^^^ 

j^o 1 Affluent: Suhnrh . This category compr i ses 

i^Ci i of :the tota^ . sample and" represents^ mdividua Is attending 
schools in a community within the city limits or residential 
area served by a city with a population greater than 150,000. 



l^Tt ^11 Percents of the total sample and, within each STOC, 
aaa to the percent representing the STOC. 

^!n^^;Z?^'^f Statistical Areas.* An economic 

a? llasi^ unit which IS metropolitan in character and contains 
mori or Jhi I °"?,?entral City with 50,000 inhabitants or 
^^o^hn-^L two cities having contiguous boundaries with a 

haS^ TnnSn?''i-^^°V^^^ ^^^^^ ^"'ailer city must 

oSnntS P°P^iftion of at least 15,000. The SMSA includes the 
county in which the central city is located, and adjacent . 

^° be metropolitan in character and 
central Sity socially integrated with the county of the 

*A few states have slightly different definitions. for SMSA. 
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They are among those ranked highest on the suburb index, 
A-r(C+D+E+F) . The communities comprising the Extreme Affluent^ 
Suburb category were located within the following three SOC 
categories: 

Big City ; 6.30%~within the city limits' of a city with a - 
population greater than 200/000. 

Urban Fringe ; 4.98% — within SMSA counties containing a 
city with a population greater than 200,000 but outside the 
city limits. 

Medium Size City ; 0.87% — within other SMSA counties not 
included in the above; ' , 

\ST0C-4f Inner City Fringe . This category comprises 8.41% 
of the total sample and replaces the Big City SOC category. It 
represents those individuals attending schools in a cbiranunity 
within the city limits of a city greater than 200,000 not 
categorized by STOC-2 or STOC-3 as Extreme Inner City or Ex- 
^i^eme; A^fiuervt ^ ^ W 

I ^ST0C-5V Suburban Frihge^ This categaryc 
of. the total sample and replaces the ^U^ 

It: represents those individuals attend sdhdol s in the 
metropolitah area, served by a city-with a pppulatioji greater than 
200, "000 :but outside rtheicity^ STOC-l, - 

STOC-2, or STOC-X as ExtriSne^^^^^^^ 
Extreme Affluent Suburb, respectively 

STOC-6, Medium City .i This -category cbmprises l^ 40% of 7 
the total s cunple and r eplace s t he Med ium- S iz e -C i ty SOC 
category. It representis those individuals^ attendihg schools 
in a city with a population between 25 , 000 and 20j3 , 000 not 
Qategorized by STOC- 1 , STOC- 2 , or STOC- 3 as * Extreme Rural , 
Extreme Inner City, or Extreme Affiuent Suburb;^ respectively. 

STOC-7, Small City . This category comprises 28.29% of the : 
total sample and replaces the Smaliier Places SOC category. It " 
represents those individuals attending schools in a community 
with a population less than 25,000 not categorized by STOC-l 
as Extreme Rural^. 

Parental Education 

This characteristic refers to the highest educational level 
attained by at least . one parent of the respondent. This infor- 
mation was provided by the respondent. For the purpose of 
definition, hicfh school refers to grade 9 through grade 12, 



18 




^ No High School. Neither parent has' any formal education 
beyond the eighth grade. . 

, Some High School. At least one parent has some formal 
education beyond the eighth grade, but neither parent "has 
graduated from high school. 

Graduated from High School . At l6ast one parent has 
graduated from high school, but neither parent has any formal 
education beyond high school. ■ 

^ Post High School. At least one parent has some formal 
education beyond high school which may include business, pro- 
fessional, or trade school training as well as college or 
university training.. 
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CHAPTER 4 



NATIONAL ASSESSMENT SAMPLES 

Overview of National Assessment Samples 

4-1. The focus of the 'National Assessment of Educational 
Progress (NAEP) is on obtaining information about the 
proportions of children^, teenagers ^ and young adults in the 
nation who respond in alternative ways to exercises in the^ 
subject areas^ assessed by NAEP— not only at one point in time^ 
but at various times so that it will be possible to determine 
what changes in knowledges f understandings rskills^ and 
attitudes are ocjcurring. This information could be obtained 
by assessing the entire population or by assessing at each point 
in time a sample carefully selected from the population so that 
it represents the entire population. Complete enumerations , in 
which the entire population would be assessed r ar^^ much more 
expensive to conduct thah^^ a the 
pbpUiation. Besides the iactor^^^^^ factors which 

fa^vpr the assessment ofe^^a^^^^ 

population include the opportunity to colrl^ct the data, over a 
I shdrter period- dfr time ^ which allows^r^^e^^ of the 

data collectedf and the^^^^^^d^^ 

highly trained and closely supervised field administration staffs 
resulting in more accurate and better gua!^^ collectipn. 
Factors such as these resulted in a decision by National Assess- 
ment planners to obtain data from representative samples selected 
from the populat^ions in whi rather than from 

the entire populations. 

Four age groups were selected to represent the populations 
of children^ teenagers and young adults : 9-year-ords / 13-year- 
olds^ 17-year-olds^ and young adults 2(5-35 years of age. In 
addition to collecting data at these four age levels /we are 
interested in obtaining similar information for certain sub- 
populations of the age populations. The subpopulat ions are 
defined by region of the country and size of the community in 
which people live^ color (Blacks and Whites) and sex. The 
definitions^ of some of these categorizations are presented in 
chapter 3; others are defined later in this chapter. 



The objective of reporting results "If or these population 
groups requires that the saraple be scientifically selected so 
that these populations are adequately represented and so that 
it is possible to draw inferences about these populations from 
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the data collected on the sample. To insure that valid infer- 
ences could be drawn, each person in the populations, to be 
sampled was given a known chance (probability) of' being included 
in our samples. 



Details of National Assessment Samples 

4.2. jrhe Interrelationship of Sample Design to 
Other Aspects of National Assessment 

National Assessment is viewed by sampling statisticians 
as a sample survey. As such> the planning of National Assess- 
ment has many features in common with the planning of all other 
sample surveys. A list of overall survey objectives must be 
prepared and all aspects of the survey design must be kept 
consistent with these objectives. The copulation or populations 
^t<r:b^^ jobse^ ^Ji^^thc^d ^f bir 

//selecting a samprle pf i^^^ be 
developed. Decisions m^st be^ m^^ about the data to b collected. 
jM6thods-of measuring ^the population members in t^^ 
/cpllecting the data must be devised ahd the fi^^ be 
prgani/ed. A plan for the summarization and analysis of data 
collecced must be prepare^^^^^^ the 
principal estimates must be given cp^ The total cost 

of .the survey must be kept within reaisonable bounds. 

The sample design is the method of selecting the members 
of the population' which are to be measured. These selected 
members of the population are called the sample. The method 
of selecting a sample from a human population can greatly 
influence the cost of locating and measuring the members of the 
sample. Alternative methods of measurement can also vary 
greatly in cost and thus limit the size of the sample if the 
total budget is limited^ The precision of estimates based on 
the sample data will be influenced both by the sample design 
and by the total size of the sample. 

It soon becomes apparent that no single feature' of a 
survey design may be developed independently of the. other 
features. The sample design^ in particular, is closely 
associated to all features of the survey design. In practice / 
these interrelationships may be explored by planning several 
alternative survey designs. Some of the alternatives may then 
be ruled unfeasible because of excessive cost or nonadherence 
to overall survey objectives. Comparisons between the remaining 
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planned methods may then be made on the basis of cost 
efficiency / expected precision of estimates^ control of bias 
in the measurement and estimations processes^ or general 
practicability. Pilot studies at this stage of the planning 
process may be employed to resolve the more difficult problems. 

The general details of sample survey planning discussed 
above apply to the planning of National As.sessment. Some of 
the specific details and results of this planning process as 
they relate' to the sample design are discussed in the remainder 
of. this chapter. ^ 

4.3. Statement of General Objectives 

For a discussion of the general objectives of National 
Assessment^ the reader is referred to chapter 1. A few of 
ttese general objectives are stated here because they directly 
affect the entire survey plan^ and, in particular ^ the sample 
design. 

1. The long range emphasis ini^ 

is to be an assessment of progress in e^^ 

2. Results of National Assessmfent should be under- 
standable to the general public. 

3. Results from National Assessment are to be used 
to describe the performance of broad population 
groups on specif ic exercises within well defined 
educational objective areas. ^ 

4 . New and different methods of collecting data were 
to be tried. 

5. No individual participant in the National Assess- 
ment survey should be required to give more than 
one hour of his time. 

In terms, of the general survey design, objective (1) above 
required that the study be repeated so that changes over^ time 
may be evaluated. The first complete cycle of National Assess- 
ment must then serve two purposes: (1) to describe the present 
status of the outcomes of the educational process; and (2) to 
establish a level or "benchmark" for future comparisons. 

Objective (2) most directly affected the development of 
reporting procedures used in National Assessment. Objective 
(3) required that the results of National Assessment be reported 
by individual exercise. Emphasis on any individual person's 
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test score in National Assessment is completely lacking. The 
Plan for summarization of collected data is to look at specific 
exercxses and summarize the responses of all individuals within 
certaxn population groups. The objective of reporting results 
by population groups was a principal constraint on the sample 
desxgn, sxnce all of these populations were to be represented in 
the sample. 

9bjective (4) most seriously affected the cost structure 
of the fxeld operations. This influence on cost subsequently 
affected the choice of an efficient sample design. 

. Objective (5) meant that each individual participating in 

Natxonal Assessment would only participate in a few of the *' 
total number of exercises assembled in a package. In terhis 
of sample design requirements, this objective meant that the 
sample desxgn should insure a representative probability sample 
for each package of exercises. ^ 

4 • 4 The Definitions of Population 

and Subpopulations ^ 

At an early stage of Nat ion al Asses sment planning , four 

age groups were selected as the target populations for 
Natxonal Assessment: 

1. 9-year olds 

2. 13-year olds 

3. 17-year olds (defined as between 16 1/2 and 17 1/2). 

and : 

V 4. Young adults 26 to 35 years of age. 

Definition of the tarcret population by age rather than year or 
grade xn school for the school age populations is one of the 
features of Natxonal Assessment that distinguishes it from most 
other educatxonal surveys and offers a particular challenge in 
sample desxgn and the organization of field procedures. 

The target population in each of these four age groups 
was Ixmxted to persons residing within the 50 states and the 
Dxstrxct of Columbia. Certain persons who live in institutions 
or Who are handxcapped were assumed to be excluded from the 
target populations. 

Initial plans were developed to categorize the population 
wxthxn each age group into subpopulations based on four 
characterxstxcs : 

1. Region of the country 

2. Type of community 

3. S3X 
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The four regions which divide National Assessment sub- 
populations are : 

1. ' Northeast 

2. Southeast 

3. Central 

4. West 



The states belonging to each of the. four National Assessment 
region subpopulatiohs are shown in chapter 3. 

Another classification of the general population into 
subpopulations that was considered important by National 
Assessment planners was based upon community characteristics. 
Four categories were considered in the planning stages: 

1. Large cities (above 180,000 population) , - - , 

:_ *2. Urt)an fringe (communitie^^^^^^ 

^ lairge citie^) > V 

3v Middle size cities (25y 000 to 180 ,000) , and 

4. Small town-rural (below 25,000) . 



4.5 The Data to be Collected - Subject 
Matter Areas 

Ten subject matter areas were included in the long range 
plans for National Assessment. These areas are Art, jpareer 
and Occupational Development, Citizenship, Literature, Mathe- 
matics, Music, Reading, Science, Social Studies, and Writing. 
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Educational objectives were developed within each area and then 
specific exercises were developed within each area to focus on 
each objective. 

. _ Due to the large number of subject matter areas, objectives 
within ar^ea, and exercises within objective , it became apparent 
that a complete cycle of National Assessment' covering all ten 
subject Tnatter . areas would be an extremely large project. Also/ 
since one of the primary objectives of National Assessment was to 
assess change or progress over time, the team assembled to 
accomplish this task would have to be disbanded after the first 
cycle and reorganized- five years later to obtain data for making 
comparisons. Such a plan, although feasible, had serious 
drawbacks. A continuous operation cycling plan involving'two or 
more subject matter areas each year was developed. This plan 
allowed for including subject matter areas in' the cycle about 
every five years. 

4.6 Methods of Data Collection - Packaging and . 

Administration of National Assessment Exercises . . 

One of the principal objectives of Natiljnal Assessment was 
to try new methods of collecting data on .educational outcomes. 
These methods were not to be limi^ted by convenience factors; for 
scoring to simple paper and pencil exercises. Each exercise 
was to be designed toward a specific objective within a single 
subject matter area; for example, a respondent should not be 
hampered on a Science exercise by his inability to read. As a 
result of this general objective. National Assessment exercises 
have many unique features. 

Most exercises aire read to the respondents. A standard 
■taped voice presentation is used for group sessions conducted - 
in schools. Other exercises require some interviewer-respondent 
interaction; these must be administered by specially-trained 
National Assessment exercise administrators. Other exercises 
can only be administered with the aid of special equipment and 
on a closely-supervised individual basis; typical exercises of 
this type are found in Science. Some of the Citizenship 
exercises used in year 01 of National Assessment involved the 
observation of discussion groups; two specially trained 
exercise administrators were required to administer and observe 
performance on these exercises. Other exercises, such as in 
the Music area, will require the recording of a respondent's 
performance on magnetic tape. 

One immediate consequence of the many unique features of 
National Assessment exercises was that specially trained 
personnel would be required to supervise and conduct the 
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administration of these exercises. The use of a centrally 
trained group of supervisors also had' the advantage of main- 
taining uniformity of administration methods in all parts of 
the country. 

^ A second principal objective of National Assessment that 
influenced the methods of data collection was the limitation of 
individual participation to one hour or less. This did not 
limit the number of exercises since, each individual respondent 
is not required to attempt all the exercises. Instead, all 
the exercises prepared for assessment of one age group, during 
one year of National Assessment are sorted into sets ofi 
exercises and assembled in what is called a "package." Ten or 
more packages may be required for each -age group. Each package 
may contain exercises from more than one subject matter area. 
A variety of types of exercises may be included. To gain 
efficiency through group administration in the school part of 
National Assessment, most of the exercise which could be 
administered to several persons assembled in a group were 
packaged in one set of packages, called group-administered 
packages . Those- exercises which ; required/ individual administra- 
tion were packaged in some limited number of individually- 
administered packages. The numbers of packages by age group and 
methbd of administration ar& illustrated for Year 02 in Exhibit 
4-1. 



Exhibit 4-1 

Numbers of National Assessment Packages 
by Age. Group and Method of Administration for Year 02 

Method of Administration 
Group Individually 
Age Level Administered Administered 

9-year olds 
13-year olds 
•17-year olds 
Young Adults 



4.7. The Need for Probability Sampling 

Most of the preceding discussion of National Assessment 
planning would be equally applicable for a complete enumera- 
tion survey of each of the target populations or for a sample 
survey of the target populations. The objectives of National 
of National Assessment might be met by a complete enumeration 



9 . 3 
13 2 
10 -2 

6 
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type survey, but the cost would be unreasonably high. The 
decennial census of population is an example of a complete 
enumeration survey of the entire population. Since the four 
target age groups are well mixed with the general population, 
the effort required for a complete enumeration National Assess- 
ment would be similar to that required to conduct the decennial 
census. . 

A sample survey based on a probability sampling method 
allows researchers- to collect data .from a small sample of the 
population and to infer from that sample certain characteristics 
of the entire population. In particular, if one is interested 
m certain population-average values, totals, or ratios, these 
values may be estimated using data collected from the sample, 
in addition, if the sampling error of the estimate can also be 
estimated from the sample data, statements about the precision 
of the estimates- can be made. A point to be stressed here is 
that sampling.. error can only show how estimates computed from 
^^J/Sof^l differ^ from the corresponding, population values; 
that would-be obtained in a complete enumeration survey using 
Identical data collection procedures. r 

Other types of errors in the estimate, such as errors due 
to nonresponse, recording errors, processing errors, or others" 
are not reflected in the sampling error . These kinds of errors 
are called non-sampling errors and could occur even in a 
complete enumeration survey. .Because a sample survey involves 
a smaller work load, more attention can .be given to supervisioh 
and training of personnel and this may in fact tend to reduce - 
the non-samplmg errors. 

Other factors besides cost limitations and control of 
non-sampling errors which favor a sample survey in general over 
a complete enumeration survey are the opportunities to collect 
data over a short period of time and the opportunity to use 
more specialized techniques because of the smaller field force 
being trained and supervised. Both of these factors are 
important in the National Assessment survey. 

Nonprobability sampling methods are sometimes used in 
conducting research projects. Samples are made up of volunteers 

^yP^°^l units"chosen on the'basis of expert judgement 
Although such methods may be adequate for certain restricted 
purposes, they do not allow any valid inference about the total 
population. Because of the importance placed on describing 
entire populations and specified subpopulations in National 
Assessment, no serious consideration was ever given to non- 
probability sampling methods. 



27 



4.8. Sample Size 



Guidelines were needed early in the planning stages on 
sample size of the number of respondents required for each 
exercise. These guidelines were to be based on a reasonable 
, level of precision for key estimates . The stated long-range 
objectives of National Assessment involve the assessment of 
progress over time. It was decided to consider the. sample size 
required to obtain sufficiently precise estimates to detect 
changes in the performance^ of a subpopulation on specific 
exercises. . . 

Initial guidelines were based on a simple random sampling 
scheme. In practicie/ when other sampling schemes must be 
used, the sample size required to achieve the same precision 
will generally be greater by a factor called the design effect. 
Exhibit 4-2 shows the required sample sizes if one wishes to 
correctly decide whether a specified change has occurred over 
time or if no improvement occurred. The sample sizes in 
Exhibit 4-2 are computied so that decision of change versus no- 
change can be made correctly 9 oiit of 10 times • 

Basea graphical interpolation of Exhibit 4-2 a 

subpopulation "effective** sample size of 400 would be adequate 
tb decide cbirectly in 9 - out of 10 cases on the average if 
positive changes of the follbwirig types have or have not 
occurred: 

1. A change of about .04 from .90 to .94 

2. A change of about .06 from .10 to .16 

3. A change of about .09 from .50 to .59 ' 



Exhibit 4-2 

"Effective "* Sample Sizes Required to Decide 
Correctly 9 out of 10 Times Whether Specified Positive 
Changes in a Proportion Have Occurred Over Time 



Specified Change from Time Level at Time 1 
Time 1 to Time 2 ^^.lO 

+ .025 
+ .05 
+ .10 
+ .15 
+ .20 

Since some of the principal target subpopulations such as 
region and community type partition the U»S. population into 



1660 

571 1311 355 

164 325 

81 142 

50 78 
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at least four groups, four times 400 or 1600 would be con- 
sidered as the appropriate effective sample size on a national 
basis to detect real positive changes over time of the type shown 
above. If the design effect were larger than 1, a somewhat 
larger sample size would be required. 

Consideration such as these led NAEP planners to consider 
sample sizes of 2000 to 2500 respondents per exercise. Since • 
exercises were packaged in several packages for each age group, 
the total number of package responses required from an age 
group was 2000 times the number of packages. As a hypothetical 
example. If 10 packages were used for each age group, the total 
number of package responses planned would be 20,000 (2,000 x 10). 
For the four age groups, the total number of package responses 

it'^nnn^%n°nnT^ ^f^""'^ National Assessment survey would be 
80,000 (20,000 X 4) . 

^ • Actual figures on planned numbers of package responses 
differ from, this hypothetical example because of varying numt,-rs 
packages by, age group and because of special adaptations of the 
sample design to the type of exercises contained in each package. 
Specific figures on planned sample sizes for year 02 of National 
Assessment are given in sections 4.11 and 4.16. 

4.9. Sampling Frames 

^ In order to select a probability sample, it is necessary 
to have a list of sampling units. Such a list is referred to 
3s a sampling frame or frame. The ultimate objeccive of the 
sampling process is to select a sample of observational u nits 
In this case , observational units are che persons in each age* 
group who are eligible to participate in National Assessment. 
In some surveys, it is possible to list all the observational 
units and m such cases, the sampling units are the same as the 
observational urits. In most surveys of human populations, it 
IS not feasible to list all the eligible participants or ' 
observational units. Some other list of sampling units may be 
much more convenient; as an example, it is much easier to com- 

school buildings- than it is to compile a list of 
q^i- ^^^1 • f ""^^ attending classes at these school buildings. 

^i^' "^^^^ simpler to compile a list of all house- ^ 

hnnlh^fr^^^K locations in a county than it is to list all 
Snn^?ni« "^^"^^^^ the county. If such units as school 
buildings or households are used as an alternative to listing 
all observational units, it is Necessary then to have a rule 

'^^^'^ identifies oach observational un!t wui 
exactly on^ sampling unit. For example, a nine-year-old may 
be associated with the school at which he regularly attends 

.'l^oft individuals can be associated with the house- 
hold of which they are a member; some specific rules are needed 
to establish what constitutes membership in a household. A 
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few persons may be excluded from one or both of these two types 
of frames because they do not go to school or do not belong 
to any household. 

In practice, further economies in listing may be achieved 
by associating households with certain identifiable land areas, 
called area segments . The area segments may be associated with 
a larger area such as a city or county. Schools may also be 
associated with the city or county in which they are located. 

Devices for constructing, sampling frames such as discussed 
above allow samplers to select the sample observational units 
by first selecting a sample of primary sampling units wnich may 
be cities or counties. Further fi -me listing then is required 
only within the selected units. Aw each stage >of sample 
selection, frames are constructed for the next stage of 
sampling only within the selected units. Eventually observa- 
tional units are listed at the final stage of sampling but only 
for a very small proportion of the total population when 
compared with a list of all the observational units in the 
target population. " ■ 

A basic issue in the planning 'stages of National Assess- 
ment was the choice of sampling frame. Several options 
considered included (1) a school frame, (2) a household frame, 
(3) a mixed household and school frame, and (4) frames based on 
other existing surveys. 

All the listed options have some disadvantages. A 
school frame clearly cannot be used to identify the young adult 
target populations. A certain proportion of 9's, 13' s, and 
17 »s also may not be enrolled in school. Since most states have 
laws requiring enrollment in school up to some minimum age, most 
9-year-olds and 13-year-olds may be expected. to be in school. 
A lower proportion of 17-year-olds may be expected to be 
enrolled. Census estimates for the 1965 noninstitutional 
population show 99.3 percent of persons 7-9 years of age 
enrolled at the beginning of the school year (October), 99.4 
percent of persons 10 - 13 years Of age, and 93.2 percent of 
persons 14 - 17 years of age.l Estimates of percent of persons 
16 1/2 to 17 1/2 enrolled vary by the time of this time of the 
year. For planning purposes, it was assumed that 75 to 80 
percent of 17-year-olds are enrolled in school. 



lu.S. Bureau of Census, Pocket Data Book, USA, 1969, p. 153. 



Since persons in all four age groups could be identified 
with a household frame, this option Sas given cons?dera?Jon 
Ir. order to identify eligible respondents in each age group 
using a household sample, a roster of household occupants by 
Sr.nnf • • "'^'^ household in the sample. Those 

persons falling m bhe defined target age populations ara then 
■ cSlL hn,^f r?^^^^^ in. National Assessment. This process is 
called household screening. Based on 1960 population estimates 
by age group and assuming a total sample in each age group 

necessary to screen 8 out of every 
10,000 households to obtain an adequate number of young adults 
h u ^° obtain the same number of 9-year-olds, 

A ?-''u°^^^^''^ ^° 50 out of every 10,000 house- 

nolds. A high proportion of the field survey costs associat^rJ 
with a household survey of this type are screening cos?s! 

be ideJJi??^/^J°^''^r? ^^^^ ^Z' 17-year-olds could 

cLif in i^i;^ - ^^^^ """^^ ""^^"^ ^ sampling frame, 

certain other economies, such as group administration of 
certain packages, could be employed with the school frame to 
further reduce costs. j-j-am^ 

school sampling frame was the 
possible loss of a large portion of the sample due to lack of 
cooperation from state or district school administration 
otticials. This factor was recognized by the NAEP planners and 
survey plans included allowances for special efforts S obLif 
tne necessary cooperation whenever at all possible. 

™on+- °^ sampling frames for National Assess- 

ment involved the use of a school frame for 9-, 13-, and 
aL7?r"°M school and a household frame for young 

adults. No effort was planned to locate 9-, and 13-year-olds 
=^^00?^^ Seventeen-year-olds not enrolled in 

vnnnS .r^f ^° s^^^ened for and identified along with the 
young adults using the household frame, but the number of 

?or^t£S vnnnS^'^r?? ""^^ ^° limited to the number required 
tor the young adult sample. 

In order to increase the number of out-of-school 
17-year-olds identified in the household screening process, 
the definition of out-of-school 17-year-olds was Lpandid (o 
include all persons 16 1/2 to 18 1/2 years old who Sere not 
enrolled in school when they were 16 1/2 to 17 1/2 years oiri 
This definition assumed that the performance of an individual 

vS.; f?\^ r^"^*^ the period of a 

Itfl that individual were not enrolled in school. Use of 

f^fi^^tion was expected to double the number of out-of- 
prScess i*^^-^tified in the household screening 
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Since the total number of out-of-school 17-year-olds 
identified in the household screening process would still be 
much smaller than a proportional allocation would demand, other 
methods of identifying such persons were considered. The use 
of other national surveys, such as the Bureau of Census, 
Current Population Survey (CPS) , as a screening device was con- 
sidered. Special procedures are required by the Bureau of 
Census to obtain permission from individuals in the CPS sample 
to permit the release of their names to an outside survey 
organization. Due to the probably poor success of these 
procedures, it was decided not to attempt the use of the CPS 
survey as an additional frame. 

In year 01 of National Assessment, no additional supple- 
mental sampling frames for out-of-school 17-year-olds were used. 
As a result, about 500 such individuals were contacted. in the 
sample. Year 02 plans called for some developmental work to 
develop and test alternative sampling- frames for out-of -school 
17-year-olds. This work is discussed fully in Section 4.19. 

4,10. Target Populations for the School Sample 

As discussed in Section 4.9, the school sample is aimed, at 
three of the four age levels: age 9, age 13, and age 17. No 
other means of sampling is usea for ages 9 and 13. Further 
attempts to identify and sample 17-year-olds who are not 
enrolled in school are made through the household sample which 
is also used primarily for adults and through special supple- 
mentary sampling frames. Both of these other sampling methods 
for age 17 are discussed in Section 4.15 to 4.19. 

The field operation for year 02 of National Assessment was 
scheduled so that each of the age levels was assessed during a 
period of approximately two months with the assessment of age 13 
beginning in October, age 9 in January, and age 17 in March. 
Students in sample schools were considered eligible to be 
selected into the National Assessment sample if their birth- 
dates fell within certain specified ranges. Thus birth date- 
eligibility requirements were based on calendar years for 
9- and 13-year olds and a special year-long range not coinciding 
with a calendar year for 17-year-olds. The same criterion for 
eligible birth date range was used throughout the survey period. 
Exhibit 4-3 shows the date of each survey and the eligible 
birth dates. Exhibit 4-4 is calculated from Exhibit 4-3 and 
shows .the extreme in the age ranges that could occur from 
application of the particular eligibility criteria. An average 
age for each age group is also shown. Based on these calcula- 
tions, the average of 9-year-olds was 9 years and 7 months^ the 
average of 13-year-olds was 13 years and 4 1/2 months, and the 
average age of 17-year-olds was 17 and 0 months. 
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Exhibit 4-3 



Specific Definitions for Year 02 
School Sample Target Populations 



Age Group Survey Period 

9 1/4/71 to 2/26/71 

13 10/12/70 to 12/11/70 

17 3/8/71 to 4/30/71 



Eligible Birth Date 

Calendar year 1961 
Calendar year 1957 
10/1/53 to 9/30/54 



Exhibit 4-4 



Age Group 
9 

13 
17 



Range of Eligible Ages Based on The 
Survey Period and Eligibility Rules 



Possible Range* 

9 years to 10 years ^ 
2 months 

12 years, 9 1/2 months 
to 13 years, 11 1/2 
months 

16 years, 5 1/2 months 
to 17 years, 7 months 



-kie 



Average Ac^e 

9 years, 
7 months 



13 years, 
4 1/2 months 



17 years 



4.11. Overall Sample Size 



Planned sample sizes for each National Assessment package 
were determined according to guidelines discussed in section 
4.8. The actual numbers also were ultimately determined by 
considering the number of primary sampling trnits and the planned 
sample size within each primary sampling unit. A sample size 
of 2,160 was planned for each individually administered package. 
The precision of estimates based on group administered packages 
was expected to suffer from a higher design effect due to more 
clustering of samples with this method of administration? 
therefore, a sample size of 2,592 per package was planned for 
the group administered packages. Exhibit 4-5 summarizes total 
sample sizes planned by age group, by package, and by package 
types. The total planned sample size for 9-year-olds was 
29,808? for 13-year-olds, 38,016? and for 17-year-olds, 30,240. 
The total planned sample size for the school sample for all 
three age groups was 98,064. Exhibit 4-6 summerizes the average 
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sample sizes by age group and by package types obtained in 
year 02* 

Exhibit 4-5 

Planned Sizes by Age Groups 
in Year 02 of National Assessment 

Age Group 

9-year-olds 13-year-olds 17-year-olds 

Number of individually 

administered packages 3 2 2 
Sample size for each 

package 2 , 160 2 , 160 2 , 160 

Total sample for 
individually adminis- 
tered packages 6,480 4,32,0 4,320 

Number of group 
administered 

packages 9 13 10 * 

Sample size for each 

package 2,592 2,592 2,592 

Total sample for 
group administered 

packages 23,328 33,696 25,920 

Total planned 

sample 29,808 38,016 30,240 

4-12 Sample Sizes Within a Primary Sampling Unit 

On a per package basis, sample sizes of 20 per package for 
individually administered packaaes and 24 per package for group 
administered packages were planned for each regular two-week 
primary sampling unit (PSU) . Each group administered package 
was to be administered to two group sessions of 12 students 
each. This general plan is illustrated in Exhibit 4-7. The 
total planned sample in a regular PSU was 276 for 9-year-olds, 
352 for 13-year-olds and 280 for 17-year-olds. The sample for 
each age varied according to the number and type of packages. 
The total for all age groups was 908. 



Exhibit 4-6 

Average Size Per Package by Age Groups 
In Year 02 of National Assessment 



Numbers of indiviually 
administered packages 

Average sample size 
per package 

Total sample for 
individually adminis- 
tered packages 

Number nf • group 

administered 

packages 

Average sample size 
per package 

Total sample for 
group administered 
packages 

Average sample size 
per .package 

Total sample for 
group administered 
packages 

Total sample size 



9-yeaf-olds 
3 

2203 
6609 



2579 

23211 
2579 

23211 
29820 



Age Group 
13-year-olds 



2199 

4398 

13 
2552 

33186 
2552 

33186 
37584 



17-year-olds 
2 

2135 

4270 

10 
2366 

23660 
2366 

23660 
27930 



In a few primary sampling units, designated as one-week 
primary sampling units, sample sizes of 10 per individually 
administered package. and 12 per group administered packages 
were planned. The details of this plan are not tabled, but the 
bottom line of Exhibit 4-7 shows the total planned samples by 
age group for a primary sampling unit. 

In terms of the field procedures required to complete 
assessment in a single PSU for a single age group, the two-week 
PSU s were scheduled for a two-week visit by a specially trained 
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National Assessment District Supervisor. During these two weeks, 
the District Supervisor assisted by two or three locally hired 
exercise administrators, completed the selection of the sample 
students and conducted all the necessary individual and group 
package administration sessions. The one-week PSUs were 
handled in a similar manner but only one week was allowed for 
completing assessment. 

Exhibit 4-7 

Planned Sample Size by Age Group 
For One Primary Sampling Unit 



Age Group 

9-year-olds • 13-year-olds 17-ygar-olds 



Niimber of individually 
administered packages 



Sample sizo for each 

package 20 20' 20 

Total sample for indi- 
vidually administered 

packages '60 40 40 

Number of group admin- 
istered packages 9 ^ 13 10 

Sample size for each 

package <v 24 24 24 

Total sample for group 

administered packages 216 312 240 

Total plcuined sample 
for one Regular or 

two-week PSU 276 352 280 

Total planned sample 
size for one- week 

PSU 138 176 140 
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4.13. General Structure of the Multi-stage 
Sample Design 

One term that can be used to describe National Assessment 
sampling is the term multi-stage. This term means that the 
sample was selected in stage. The advantages of multi-stage 
nn?^?n%oi^?«r!^a°^ to Sampling frame development were pointed 
out m Section 4.9. Multi-stage designs can also be used to 
concentrate or cluster the sample and thus reduce field costs. 
In order to discuss multi-stage designs, it is necessary to 
speak about several types of sampling units; namely, primary 
or first-stage sampling units. Secondary or second-stage 
sampling units, third-stage sampling units, and so on. 

Four stages of sampling may be identified in the 
National Assessment school sample design. The primary sampling 
units were geographic land areas consisting of one or more whole 
counties The sample selected consisted of- 116 primary 
sampling selected with probabilities proportional to an estimate 
of size. The estimated size measure was the number of 17-vear- 
olds in each^primary sampling unit. The principle of selecting 
sampling units with probabilities proportional to their size or 
their estimated sizes was used at all stages of sampling. 

The same primary sampling units were used for all three 
target age group samples. Secondary sampling units or second- 
stage units were formed within primary sampling units by 
grouping schools by zip code areas, in most cases several zip 
code areas were used to form one secondary sampling unit; 
hfSh?^^''^ sampling units of this type were employed only in the 
highly populated primary sampling units; in very small primary 
sampling units, this stage of sampling was not necessary. It 
was hoped that this type of clustering, produced by fontiing 
secondary sampling units within the large PSUs would simplify ' 
field procedures and ease the inconveniences to the local school 
systems by reducing the number of different administrative units 
involved. Secondary sampling units were also used as means of 
insuring that some relatively high socio-economic areas and 
f™i a^eas were included in the 

primary sampling unit. In most cases, two 
l?ocetees^lS? ^2?.:;?^'^^ ^^^^ selected per primary sampling unit. 
aua^n^^r^-hSf secondary sampling units did not 

fo? S^th^ee leSlls!"^ secondary sampling units would be used 

ahiii*-v^™^?i'''^^^^^P sampling units were schools. A prob- 
aroun f ro^^io ^f^^^^fools was selected indepandly for each age 
group from the schools m the secondary sampling unit selected 
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for the age group. The number of schools to be selected in 
each second stage sampling unit was not, fixed ^ but was deter- 
mined by the approximate number of students in the eligible 
age level attending each school. In other words ^ if the 
schools had very low enrollment more schools were required in 
the sample in order to obtain the prescribed number of students 
for participation in National Assessment. A general guideline 
imposed on the sample design at this point was that at least 
two schools be selected from each secondary sampling unit so 
that no single school would be required to supply a large 
proportion of the respondents within the PSU. 

The final stage of sampling was the selection of a 
probability sample of students from the eligible age group at 
each sample school. The fourth stage units were then the 
students who were eligible to participate by the birth date 
rules. As students were selected it was also necessary to 
assign a specific package to each student. Special procedures 
were developed for selecting sample students and assigning 
packages which allowed some packages to be Administered in . 
group sessions of 12 students per session and other packages 
to be administered on an individual basis." These procedures 
also insured that the students participating in any particular 
package were a probability subsample of all the students 
participating in National Assessment. 

4.14. Special Requirement Imposed on the 
School Sample Design 

One change that occurred between year 01 and year 02 of 
National Assessment was the addition of the requirement that 
all states be included in the sample. This requirement did not 
in any way imply that the sample in each state should be ade- 
quate for reliable state estimates. 

This requirement meant that if at least one two-week 
primary sampling unit was selected from each state, certain 
states with low total populations would have a larger proportion 
of the total sample, thus would be prescribed by allocation 
proportional to population. Sampling schemes that deviate 
greatly from a proportional allocation will generally produce 
estimates for the total population with a lowered precision per 
dollar. The use of one-week PSUs in these small states helped 
to alleviate the problem of disproportionate allocation. 

4. 15 Target Populations for the 
Household Sample 

As discussed in section 4.9, the household sample was 
aimed at two of the four age groups: 17-year-olds not enrolled 
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m school and young adults 26 to 35 years of age. No other 
means of sampling was used for the young adult population. 
Approximately 90 to 91 percent of the 17-year population were 
enrolled m elementary and secondary schools. These eligibles 
were identified and assessed through the school sample. The 
household sample and the supplementary frames were used to 
sample the remaining 17-year-olds who were not enrolled in 
school. 

The field operation for year 02 of National Assessment 
was scheduled so that the young adults and out-of-school 
17 -year-olds were assessed during a period of approximately 
five months beginning in March, 1971 and ending in July, 1971. 
The majority of eligibles sampled through the household ' survey 
were assessed during April, May, June and July. Most of the 
out-of-school 17-year-olds sampled through special frames were 
assessed during June and July. Sample persons were considered 
eligible if their birth dates fell within certain specified 
ranges. In addition to young adults and out-of-school 17-year- 
olds, an additional age group was assessed. This group was 
defined as 18-year-olds who were not enrolled in school when 
they were 17-year-olds, while this group was not part of the 
population of 17-year-olds; it was assumed that they would 
perform essentially the same way on National Assessment 
exercises as the out-of-school 17-year olds. Exhibit 4-8 
shows the survey period and the eligible birth dates for the 
^fr5v,^°^'' . "^^^ period coincides approximately 

EiMb^?%^r''^Jh^^''t-'^-J?'' in-school 17-year-old assessment 
Exhibit 4-3. The eligible birth dates for out-of-school 
17-year-olds (Group A) are identical with the eligible dates 

t°''*.u ^T""'^ .^^^^^^^^ school sample. The eligible 

birth dates for 18-year-olds (Group B) are exactly one year 

tt Exhibit 4-9 is calculated from 

the Exhibit 4-8 information and shows the extremes in actual 
ages which could occur using the eligibility criteria. An 
average age for each age group is also shown. The averaqe aae 
for young adults was 31 years, 4 1/2 months, based on these 

o°f.^^^ ^"'^ 18-year-olds, the average ages 
SJpli ^r""^' ^ ^i^^.u°"^^^ y^^^^' 2 1/2 months; respec- 

^i^?,H^ 18-year-olds include only 

individuals who were not enrolled in school when they were 
approximately 17 years of age (see footnotes to Table 4-9) . 

4.16. Overall Sample Size 

h^^e^v,^?^^""®'^ sample sizes for the year 02 National Assessment 
household survey are shown in Table 4-10. The planned sample 
size for each package is smaller than 2,000 to 2,500 determined 
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in early project planning and discussed earlier in Secion 4. 8* 
The reasons for the smaller planned sample size necessitate 
reference to two previous surveys - the year 01 National 
Assessment household survey and the subsequent quality check 
survey. The year 01 household survey results did not approach 
the completeness or quality of the year 01 in-school assessment. 
In summary r a household screening rate of 77% and an individual 
completion rate of 57% were achieved. Thus, the overall rate 
was approximately 44%. The quality check survey results 
indicated that more experienced interviewers^ experienced con- 
stant supervision and monetary incentives to respondents could 
raise the overall completion rate for young adults to over 
70% (91-1/2 percent household screening completion and^ with the 
incentives^ 80 percent . completion rate for eligibles in screened 
households). To test these results r the year 02 planned" sample 
sizes were set at approximately one-half those which might be 
regarded as full-scale operational sample sizes for National 
Assessment. The household sample was limited to ^the number of 
households required to produce approximately 1<>^00 responses for 
each of the six young adult packages. Out-dfcschool 17-year- 
olds were screened and assessed using the household frame/ but 
the bulk of the out-of-school 17-year-old resppndent^ were 
assessed from supplemental list frame surveys. ^*hrefe supple- 
mental frames were sampled in year 02 of National Assessment. 
About two-thirds of the sampling was from lists of high school 
dropouts provided by a sample of schools. Enrollees at Job 
Corps and Neighborhood Youth Corps sites were sampled also; 
these made up approximately one-third of the supplemental list 
sampling for out-of-school 17-year-olds. 

All packages were administered individually to young adult 
and out-of-school 17-year-old respondents in the household 
survey. In addition ^ each respondent was given the option of 
completing one^ twor three or four assessment packages. It was 
decided to offer an incentive of $5.00 per package to the 
respondents agreeing to complete two or more packages / based 
upon the results of the year 01 household survey quality check* 

For the supplemental frame surveys ^ all packages were 
administered individually to 17-year-old respondents from the 
school dropout list sample. These respondents were also given 
the option of completing up to four assessment packages per 
respondent with the same monetary incentives offered household 
survey respondents. Some of the packages were administered to 
groups of eligibles from the Job Corps and Neighborhoold Youth 
Corps samples. Exhibit 4-11 shows the planned sanple sizes 
for group and individual packages in all three of the supple- 
mental frame surveys. 
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Exhibit 4-8 

Specific Definitions for Year 02 ^ 
Household Sample Target Populations 



Age Group 

Young Adults 
26 to 35 

17 Year Olds 
Group a1 
Group b2 



Survey Period 
3/1/71 to 7/31/71 



3/1/71 to 7/31/71 
3/1/71 to 7/31/71 



Eligible Birth Dates 
4/1/35 to 3/31/45 



10/1/53 to 9/30/54 
10/1/52 to 9/30/53 



^Not enrolled in school 1/1/71 to 1/31/71 
2 

""Not enrolled in school 3/1/70 to 3/31/70 



Exhibit 4-9 

Range of Eligible Ages Based on Survey 
Period and Elibibility Rules 



Age Group 



Young Adults 
26 to 35 



17 Year Olds 
Group a1 



Group B^ 



Possible Range 

25 years, 11 months 
to 

36 years, 4 months 



16 years, 7 months 

to 

17 years, 10 months 

17 years, 7 months 

to 

18 years, 10 months 



Average Age 

31 years, 1 1/2 
months 



17 years, 2 1/2 
months 



18 years, 2 1/2 
months 



•'•Not enrolled in school when 16 years, 3 months to 17 years, 
4 months of age. 

2 

Not enrolled in school when 16 years, 5 months to 17 years, 
6 months of age. 
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Exhibit 4-10 



Sample Sizes for Year 02 National 
Assessment Household Survey^ by Age Groups 

Age Group 

Estimated Number of : Young Adults Out-of-School 17 's 
Households 6,925 6,925 
Eligibles 2,547 76 
Respondents 1,783 53 
Completed packages 6,240 185 
Number of packages 6 12 
Planned Sample Size 

for each package 1,040 15,4 

Altered Average 
Sample Size per 

Package . 1,244 27 



Each of the respondents in the Neighborhood Youth Corps 
and Job Corps samples were asked to complete two packages each - 
either two group packages or the two individually administered 
packages. An incentive of $10.00 per respondent was offered. 

4.17. Sample Size with a Primary Sampling Unit 

On a per-package basis, each young adult package was 
expected to be administered approximately 20 times in each 
primary sampling unit (PSU) in. the household survey. It should be 
noted that the planned sample sizes per PSU are averages for 
the household sample rather than fixed numbers, as in the school 
sample. Exhibit 4-12 illustrates the average number of house- 
holds, eligibles, respondents and completed packages for the 
year 02 household survey. 



Within each PSU, a National Assessment Field Supervisor 
visited the PSU initially to list several sample segments and 
interview potential field interviewers. The Field Supervisor 
made additional periodic visits to list additional segments, 
assist the field interviewers with probJems, conduct quality 
check work and give further training to the field interviewers. 
There were a total of 52 PSUs in the year 02 household sample. 
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Exhibit 4-11 



Planned Sample Sizes for Year 02 National Assesment 
of Out-of-School 17-Year-Olds from Supplemental Frames 

Supplemental Frames 



Number of individually 
administered packc.ges 

Sample size for each 
package 

Total sample size for • 
individually adminis- 
tered packages 

Number of group adminis- 
tered packages 

Sample size for each 
package 

Total sample size for 
group administered 
packages 

Total planned sample size: 

Completed packages 
Number of respondents 



School Job 
Dropout Corps 
Lists Sites 



12 



102 



1,225 



1,225 
350 



20 



40 



10 



24 



240 



280 
140 



Neighborhood 
Youth Corps 
Sites 



20 



40 



10 



24 



240 



280 
140 
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Exhibit 4-12 



Planned Average Sample Size by Age Group for One Primary 
Sampling Unit Year 02 National Assessment Household Survey 



Age Group 

Estimated Number of ; Young Adults Out-of -School 17' s 



Households 133 133 

Eligibles 49.0 1.5 

Respondents 34 . 3 1,0 

Completed Packages 120 "3.6 

Number of Packages 6 12 

Sample Size for each 

package 2 0 - .30 



Each PSU contained ten sample seg^'ents of approximately 13 
occupied housing units each. 

The supplemental frame samples were specially designed to 
survey out~of -school 17-year-olds only. The year 02 National 
Assessment school sample, wcj subsampled as one of the supple- 
mental samples. . Lists of Job Corps sites and Neighborhood 
Youth Corps sites were also sampled as supplemental frames. The 
selection of samples from these supplemental frames is also 
discussed in later chapters. Exhibit 4-13 shows planned 
sampled sizes per PSU for the supplemental frames used in the 
year 02 National Assessment. 

4.18. Structure of the Household Sample Design 

The National Assessment household sample was designed as 
a multistage sample. The advantages of multistage designs were 
discussed in section 4.9. 

The primary Sampling units were geographic land areas 
consisting of one or more counties. The sample consisted of 
'52 PSUs selected with probabilities proportional to a measure 
of size. The size measures used were in the 1970 census pre- 
liminary county populations. 

V ^ 
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The secondar y sampling units were land area segments 
or clusters of housing units within the primary sampling units. 
The area segments were defined to contain an average of approxi- 
mately 13.3 occupied housing units and 4.9 eligible young 
adults. The secondary sampling units wi'-hin each PSU wer6 
stratified into five socio-economic status substrata and two 
sampling units were selected with equal probabilities from each 
substratum. 

The third-stage sampling units were the occupied housing 
units within the secondary sampling units and the fourth-stage 
sampling units were eligible young adults and out-of-school ' 
17-year-olds within the occupied housing units. There was no 
subsampling within the secondary units; that is, all eligibles 
in all occupied housing units of the secondary sampling units 
were designated as "in the sample." 

4.19. Structure of the Supplemental 
Frame Sample Designs 

Multistage sampling designs were also used to select the 
samples of out-of-school 17-year-olds from supplemental frames. 

As mentioned in section 4.17/ a subsample of sample 
P5US and sample schools selected for the year 02 school sample 
were used as the basic sample for the school dropout sample. 
Thus, the primary sampling units , secondary sampling units . 

third-stage sampling units"~ useu tor trie school drnpnn^ 
sample were those discussed in section 3 of this report. 
One-half of the year 02 PSUs and one-half of the schools in the 
sample for the 17-yeat-old in-school assessment in those PSUs 
were selected for the school dropout ' sample. The fourth st age 
sampling units were high school dropouts on lists furnished by 
the sample schools. The lists of dropouts contained some 
individuals .not meeting the National Assessment eligibility 
criteria shown in Exhibit 4-8. 

Two-stage sample designs were used for the Job Corps and 
Neighborhood Youth Corps supplemental frame samples, which wer^ 
selected independently. The first-stage sampling units in both 
cases were Job Corps sites and Neighborhood Youth Corps' Centers 
Identified on lists obtained from U.S. Department of Labor. 
Primary samples consisting of five Job Corps sites and five' 
Neighborhood Youth Corps centers were selected using probabili- 
ties proportional to an estimated size measure. The measures 
of size used were the site and center capacities, in terms of • 
number of enrollees. 



The second-stage sampling, units were enrollees at the 
Job Corps and Neighborhood Youth Corps ESUs who ^ were eligible 
for National Assessment* Eligibles were identified from 
records available at the PSUs. Equal-sized samples of eligibles 
from each ^site were selected from these lists of eligibles with 
equal probabilities. 

4.20. Special Requirements Imposed on the 
Household Sample Design 

As mentioned in section 4.16, the vear.02 National 
Assessment household sample size was approximately one-half what 
would be considered the fulL-scale National Assessment sample 
size. Problems occurred in the year 01 household survey which 
led to a decision to use a smaller sample in year 02 to test 
whether modifications in the field procedures would lead to 
more acceptable results. It was expected that the year 03 
household sample would be expanded to the operational level of 
2,000 to 2,500 responses for each young adult package, assuming 
that the response rates using the modified field procedures 
were acceptable. Thus, for reasons of economy, the primary 
sample used in year 02 was required to be designed as an 
expandable sample. That is, a larger primary sample might be 
required for the year 02 household sample and it would be more 
efficient, cost-wio3, to have the larger year 03 primary sample 
of PSUs include the 52 PSUs selected for the year 02 primary 
sample. This requirement was met by selecting 208 PSUs with 
probabilities proportional to size and subsequently selecting 
52 of the 208 PSUs with equal probabilities for the year 02 
household survey primary sample. Additional PSUs could be 
selected from the 156 remaining PSUs for use in year 03 and the 
overall probabilities of selection for PSUs in the expanded 
sample would still be proportional to size. 

Future plans included the possibility that the same 
primary • sample might be used for several years. Thus, PSUs 
were required to contain sufficient respondents for several 
years of household assessment without assessing any respondents 
more than once. This requirement was met at the time the 
sampling frame was constructed by requiring that each' PSU in ' 
the frame must include at least 20,000 population in 1970. 
Thus, counties with 1970 populations less than this minimum 
were combined with other counties prior to the primary sample 
selection. Another reason for establishing a minimum size for 
PSUs was the policy of oversampling the low-SES portion of the 
population. Since the oversampling was to be effected within 
sample PSUs for certain size of community strata, it was essen- 
tial that all PSUs contain a large enough population to make 
oversampling feasible. 
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Unlike the National Assessment school sample, there was 
no requirement that all states and District of Columbia be 
included in the primary sample for the household survey. Conse- 
quently, no such restriction was imposed on the sample design. 



Exhibit 4-13 

Planned Sample Size for One Primary Sampling 
Unit for Year 02 National Assessment of Out-of-School 
17-Year-Olds from Supplemental Frames 



Supplemental Frames 



Number of individually 
administered packages 

Sample size for each 
package 

Total sample size for 
individually adminis- 
tered packages 

Number of group adminis- 
tered packages (per PSU) 

Sample size for each 
package 

Total, sample size for 
group administered 
packages 

Total planned sample for 
one PSU: 

Completed packages 
Number of respondents 



School Job 
Dropout Corps 
Lists Sites 



12 



1.76 



21.1 



8 

4I 
12 

48 



Neighborhood 
Youth Corps 
Site 



8 

4I 
12 

48 



21.1 56 
6.0 28 



56 
28 



^^^^u°^Jr^^ ^^°"P administered packages was assigned to two 
of the five Job Corps PSUs and to two of the five Neighborhood 
Youth Corps PSUs. 
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CHAPTER 5 



ADMINISTERING THE ASSESSMENT 

In this chapter, we discuss the procedures involved in the 
actual collecting of the data which, when processed, tell us 
what knowledges, Understandings, skills, and attitudes various 
National Assessment groups! have attained in the 10 assessed 
subject areas. 2 

Most individuals wno participate in the National Assessment 
project are enrolled in school at the time. This "in-school" 
sample includes all 9-year-olds, all 13-year-olds, and most 
17-year-olds. 3 

In addition to .assessing children and teen-agers enrolled 
in school, we assess two groups of individuals who are not 
enrolled in school. This "out-of-school" assessment includes: 
(1) adults aged 26-35 and (2) those 17-year-olds who have 
either dropped out of secondary school or who have completed 
secondary school at an early age and may or may not be enrolled 
in college or its equivalent. 4 The adult assessment provides 
information about the current knowledges, understandings, skills, 
and attitudes in the 10 assessment subject areas possessed by . 
individuals most of whom have been away from formal education for 
a number Qf years. The out-of-school 17-year-olds are included 

in the assessment to make the results for 17-year-olds-more 

representative of all 17-year-olds. 

Since the details of the in-school assessment and the out- 
of-school assessment differ considerably, they are discussed in 
separate sections. The discussion is oriented to the year 02 
assessment of Reading and Literature, but many of the procedures 
can be generalized to other assessment years. 



See chapter 3 for definitions of the National Assessment groups. 
See chapter 1 for a listing of the 10 subject areas. 
See chapter 3 for definitions of these age levels. 

See chapter 3 for a definition of the adult and the out-of- 
school 17-year-old populations. 
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The In-School Assessment 

The National Assessment staff working on the in-school 
assessment was concerned with two major activities — placing the 
selected National Assessment exercises into administrative 
units called packages, or booklets, and administering the 
packages to 9-, 13-, and 17-year-old students in accordance 
with the in-school sampling plan (see chapter 4) • The actual 
compilation of packages was done by the Research Triangle 
Institute (RTI) of Raleigh, North Carolina; the printing of the 
packages was done by the Measurement Research Center (MRC) of 
Iowa City, Iowa; and the field administration was conducted by 
RTI with a subcontract to MRC for the Central and Western 
regions. The National Assessment staff performed planning, 
coordinating, and monitoring functions for all administration 
processes. 

At each age level, the selected exercises were packaged 
into 35 minute units. Most packages were designed to be 
administered to students in groups of 12. Some packages, how 
ever, were designed to be administered to individual students in 
ah interview situation. For age 9, there were nine group- 
administered packages and three individual-administered packages; 
for age 13, there were 13 group packages and two individual 
packages; and for age 17, there were 10 group packages and two 
individual packages. The National Assessment packages were 
constructed according to these criteria: 

1. Each package contained exercises from both Reading 
and Literature. 

2. Each package contained exercises from the three 
levels. of difficulty (easy, medium, and hard) and 
began with an "easy" exercise. 

3. Some exercises could not appear in the same 
package due to similarity, keying, etc. 

4. Exercises could not appear in the same package if 
they used the same, or similar, stimulus materials. 

For each group package, a tape recording was produced. The 
tape first gave all introductory statements. Sample exercises 
were on the tape as well as printed in the booklet. .For Litera- 
ture, all exercise directions, written stimulus materials, 
exercise stems, and exercise foils were both on the tape and in 
the printed booklet. For Reading exercises which were dependent 
upon reading ability, only the exercise directions were on the 
tape. The tape assured uniformity of administration by different 



exercise administrators and by the same exercise administrator 
from one administration to another. The tape also helped to 
prevent exercises for subject areas other than Reading from 
measuring reading ability instead of the subject area in 
question (e.g., Literature for year 02). The tape' was paced to 
allow the proper amount of response time for each exercise. 
Students marked their answers directly into the packages. 

In individual packages, directions were given by the inter- 
viewer to the student being interviewed. Poems or stories 
used as Literature stimuli were read to the student from a tape 
recording to assure uniformity in the stimulus materials. In 
most cases the interviewer wrote the student's oral remarks in 
the packages; however, the packages were handed to the students 
for certain exercises. The student responded directly into a 
tape recorder for one Literature exercise. 

Thirty-five minutes of— exercise-time-per-package-aHowed-^ 
the field staff, to work generally within class periods of 
schools and to avoid using students for more than approximately 
50 minutes. Introductory statements, completion of personal 
data questions, tape time not associated with exercises (in 
group packages) and miscellaneous administrative time generally 
required about 15 minutes per package administration. 

Before the packages could be administered in selected 
schools, cooperation had to be obtained from school personnel 
and operational procedures had to be established between them 
and the National Assessment staff. Chief state school officers 
were informed that identification of all schools selected for 
the assessment within their respective states would be available 
upon request. Many states requested this information, and some 
then contacted superintendents to tell them schools from their 
districts had been selected for participation -in the project. 
The first contact by National Assessment was a letter in late 
July, 1970, from the Staff Director. In addition to being 
notified of the selection of his schools in the sample, the 
superintendent or private school official was told he would 
receive descriptive materials from RTI or MRC, depending upon 
the geographical region. These materials informed the super- 
intendent of the identity of all his schools selected for the 
year's assessment, gave the dates for all contact with him, his 
principals and other school personnel, described National 
Assessment, and informed him that a member of the field staff 
would telephone to arrange a meeting with him and the principals 
of schools selected for 9- and 13-year-old assessments. A 
proposed date for the meeting was included in the letter. 
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The Research Triangle Institute and the Measurement 
Research Center employed four Regional Supervisors (one for 
each of the geographical regions) and 29 District Supervisors. 
The District Supervisors were responsible for arrangina the 
meetings with superintendents and -principals , setting assessment 
dates, and conducting the actual assessment. 

During late August, each District Supervisor called the 
superintendents in his district to answer questions 
about the project, confirm dates. about selected schools, and 
arrange the meeting for the proposed date, or another mutually 
convenient date, m September or October. At the meeting the 

f^Pl^i^ed the purpose of National Assessment, 
described operational procedures for completion of fhe assess- 
ment, and determined weeks suitable for assessment in the area 
In most areas, several school districts were to be involved 
so the District Supervisor suggestad assessment weeks to be 
^greed upon by school personnel at the various meetings. 
Assessment of 13-year-olds was arranged for October, November 
and December, 1970, and 9-year-old assessment was arranged foi 
January and February, 1971. 

h.H principal appointed a coordinator, a staff member who 

had a flexible schedule, to prepare for the assessment. Prior 
to assessment week the coordinator arranged to have a Student 
Listing Form completed for each student born during the calendar 
^u^^^ f^"-""-"^ National Assessment eligibility (see chapter 3 for 
the definition of age levels) . The District Supervisor used 
these cards to randomly determine the students to be assessed 
iH.S^^ school (see chapter 4) and to obtain the birthdate, sex, 
grade, and color for each. When the District Supervisor 
arrived during assessment week the coordinator also arranged for 
space to conduct the group and individual administrations and 
worked with the District Supervisor to arrange a mutually con- 
venient schedule in the school and to insure that students 
arrived on time for their scheduled administrations. 

For 17-year-old assessment, the introductory meetings were 
held concurrent with the assessment of 9-year-olds, and the 
assessment was conducted during March and April, 1971. 

After the sample was selected in a school, package admin- 
istration was completed by the District Supervisor or by an 
Exercise Administrator who was hired locally and trained by the ^ 
District Supervisor to professionally administer National 
iJoM^^r"^ packages. Exercise Administrators had various back- 
grounds including teaching, substitute teaching and survey 
research. Assessment time. varied in schools dependinq noon thf> 
?no%°days assigned, but it rarely lasSd morl S 
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As packages were administered, the District Supervisor 
checked all data coded on the package from Student Listing 
Forms for accuracy. When the District Supervisor completed 
assessment in a school, he instructed the coordinator to save 
these forms for six months in case they were needed for data 
verification.. Since names were not associated with National 
Assessment packages, these cards which had been matched with a 
package by ID number were the only means of verifying lost or 
questionable data. After six months the cards were destroyed 
to assure the anonymity of students who participated in the 
assessment. 

The District Supervisor mailed the packages to the scoring 
contractor where a quality check was performed to minimize 
errors in the data. Each entry from the Student Listing Form 
was both coded in arabic numerals and gridded for optical 
scanning. The codes and grids were checked against each other, 
and both were checked against acceptable ranges for the various 
entries. Errors which could not be reconciled by central staff 
were sent back to the District Supervisor to be checked against 
the Student Listing Forms. 

Other quality checks included a small sample of schools 
from which information from the Student Listing Forms was 
obtained by central staff to be checked against packages 
returned from the field. This was to check the accuracy of 
transferring information recorded by the schools. Also, the 
central staff member discussed overall assessment procedures 
with the principal and coordinator in this sample to discover 
problems with field procedures and implementation. Each school 
which participated in the assessment received a thank-you letter 
and a questionnaire concerning contact procedures ^ preparation 
for assessment/ assessment exercises, personal data questions, 
and the involvement during assessment week. The information 
was analyzed to discover problems in the field procedures which 
required modification for future assessments. 



Out-of-School Assessment 

The National Assessment staff working on the out-of-school 
assessment was concerned with placing the selected exercises 
into packages and administering the packages to the adults and 
out-of-school 17-year-olds in accordance with the out-of- 
school sampling plan (see chapter 4) The actual compilation 
and printing of the packages was done by the same contractors 
who performed these functions for the in-school assessment — 
Research Triangle Institute (RTI) and Measurement Research 
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Center (MRC) , respectively. The field administration of the 
packages to the individuals in the out-of-school sample was 
handled exclusively by RTI and occured from March to August, 
^ri'^ As with the in-'school assessment, the National Assessment 
staff performed planning, coordinating, and monitoring functions 
for all administration processes. Packages for the out-of- 
school assessment were constructed according to the same 
criteria as the in-school packages. In fact, the packages used 
m assessing the out-of-school 17-year-olds were the same as 
those used in assessing the in-school 17-year-olds. 

Obtaining out-of-school individuals within the appropriate 
age ranges was more difficult than obtaining in-school students 
The Field Supervisors personally identified and inspected the 
area segments to be sampled and .compiled a list of all the 
sample housing units that fell within the area segment 
boundaries. 

The Field Interviewers canvassed the sample housing units 
for eligible respondents in the two out-of-school age 
categories. They filled out a screening questionnaire on which 
they listed all household members and their exact birthdates. 
In this way, the screening process had the appearance of a * : 
simple survey. The interview was terminated if the screening ^ 
questionnaire listed no eligible household members. Using a ^ 
callback procedure for households where no one was at home, the 1 

oi! 1!^^''''''^''?''^ completed screening questionnaires on over . I 
99% of the sample housing units. • j 

The Field Interviewers then solicited cooperation in taking I 
the assessment from those individual s" who were deemed eligible I 
on the basis of the screening questionnaires. The eligible 1 
individuals were offered a monetary incentive to complete more 'i 
than one package: $10.00 for two packages, $15.00 for three ? 
packages, and $20.00 for a maximum of four packages completed. i 
No monetary. compensation was made for completing only one ' 
package. The four packages given to an adult or out-of-school 
17-year-old were selected such that all packages within each 
age level were administered to approximately the same number of 
individuals and such that each package was administered in each 
ordinal position (first, second, third, or fourth> about the 
same number of times. Field interviewers were equipped with 
various National Assessment handouts in case a respondent 
wanted additional information about the project. If a coopera- 

7imL- i^t^-'-^.u^'^^Y^*^"^^ taking the assessment I 

immediately, the Field Interviewer made an appointment to come 1 

back at a later time. The eligible individuals who refused to I 

respond were noted by the Field Interviewer, and these cases i 
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were reviewed by the Field Supervisor. In as many cases as 
possible, the Field Supervisor contacted those who refused in 
an attempt to convert the refusal to a participating respondent* 
This procedure produced successful results in many instances. 

Although the out-of-school assessment is always a one-to- 
one situation (interviewer and respondent) / the administration 
of the packages is kept comparable to the in-school mode of 
administration. The tape was not paced, however. The inter- 
viewer started and stopped the tape as necessary to allow the 
respondent a reasonable length of time to respond. 

For the year 02 assessment of Reading and Literature, 75% 
of the eligible adults .and 97% of the out-of-school 17-year- 
olds contacted consented to take at least one package. The 
average number of packages taken per respondents was 3.9. 

The Field Supervisors were responsible for frequent contact 
with the Field Interviewers, both by personal visit and phone 
contact, to maintain a high degree of field quality and 
proficiency* The Field Supervisors revisited a sample of the 
housing units to validate the interviewers' work. RTI also 
used a mail-validation procedure to further safeguard field 
quality* Completed packages and screening forms were subject 
to quality checks upon being received at RTI, where field 
errors were recorded and brought to the attention of the field 
staff. 
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CHAPTER 6 



SCORING OP EXERCISES 



Each National Assessment exercise is scored in one of 
three ways: by a subject matter specialist (Professional scoring) 

.by a well trained scorer (Semi-Prof essional) or by a machine. ' 

Professional scoring is used for those items that demand 

difficult analyses, informed value decisions, or expertise in ; 

a particular subject area. Primarily, Professional scoring is j 
necessary for the evaluation of essays, the subjective analysis 

of written material , or the evaluation of performance exercises- < 
-that is, exercises which require the respondent to perform in 

some prescribed manner best judged by an expert. c ] 

Exercises that are professionally scored may or may not T 

require that the staff of subject area specialists be repre- ] 
sentative of - various geographic regions and/or other 
considerations, but when geographic diversity is desirable, then 

Professional scoring is normally conducted in a work seminar at ] 

a central location. If, however, the only requirement is sub- : 

ject area expertise, then the staff is assembled from one ^ 

locale and remain there for the scoring of exercises. ', 

i- 
? 

The semi-professional scoring is used for the scoring of i 

items that contain short answers (such as one-word or title \ 

responses), brief essay responses, and listings or other j 

comparable answers that are not as difficult or complex as those - 

requiring professional scoring. There may also be a number of \ 

exercises that can be scored on a right-wrong basis depending ! 

upon the content of the response. In these instances the staff \ 

requirements are somewhat less than those required for \ 
professional scoring but follow the same pattern. 

! 

The third type of scoring, machine scoring, takes advantage I 

of optical scanning devices that automatically score multiple- i 
choice responses without the need lor any individuals. This 

method provides a fast economical and practical way to get 5 

results into the computer system for analysis. Ultimately, all ^ 

scores even the professional and semi-professional results are 1 

translaued into terms that can be optically scanned by machine i 

and fed irto the computer system for faster analysis. ^ 

The educational requirements and experience for the I 

professional and semi-professional staff are somewhat similar, \ 
but do vary in the degree of formal training and experience 
that are brought to the project: 
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Professional scoring staff requirements . 



1. Professional .>corers hold an advanced degree 
in the subject area under consideration* They 
either have a Ph,D, or are engaged in a Ph.D. 
program. 

2. Professional scorers have actual experience in 
their major field and are able to bring to the 
program the results of this experience. If at 
all possible, this includes actual experience 
with the age groups under consideration in the 
National Assessment program, 

3. All Professional scorers are approved by the 
National Assessment staff before they are 
included in the project. 



Semi-professional scoring staff requirements . 

1. Semi-professional scorers have a degree in the 
subject area field in which they are working. 

2. Semi-professional scores have some actual 
experience in the field in their subject area 
and are able to bring the results of this 
experience to the project. 

3. All semi-professional scorers meet' the require- 
ments of the specifications set forth by the 
National Assessment staff, and are approved by 
National Assessment before they are included in 
the scoring program. 

Once a staff whether Professional or Semi-Prof essional/ 
has been assembled for the scoring of the exercises for rne 
National Assessment program, a formal training program is 
conducted to acquaint the staff with the requirements of the 
project. This formal training program consists of three 
basic steps which are necessary for the pr^-per scoring. These 
steps are as follows: (1) development of standards and guides 
for scoring, (2) the use of "live data" to verify the standards 
and guides and to provide actual scoring experience, (3) the 
resolution of discrepancies. The first step — the development 
of standard and guides — involves establishing the scoring 
criteria that are used and determining the type of information 
that is necessary for further reporting. These, then, act 
as the guides used by the scoring staff in the final analysis 
of the responses given by the participants. 
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The second step in the process, the use of live data, is 
the actual training of the staff in scoring techniques. During 
this phase, the scorers work with actual tryout data from the 
^ field, and the results of their initial scorings are checked 
carefully to determine that scorers are, in fact, correctly 
utilizing the standards set forth for- the exercises. In 
addition, it gives the scoring staff the opportunity to adjust 
or modify the standards and guides wherever necessary, based on 
the results of their initial scoring of the live data. 

The third and final step, the resolution of discrepancies, 
may take one of two forms. In the case of professional scoring, 
if the requirements for the subject area state that more than 
one reading or scoring of a given exercise is needed, then 
the resolution will be based on a second reading of the exercise 
and/or the comparative reading of a referee. 

The second form is that of a quality check, which is a 
normal, ongoing procedure regardless of whether or not the 
exercises are professionally scored or semi-prof essionally 
scored. This consists of a check of the work of each' of the 
scorers on a periodic basis tb determine that there are no 
discrejpancies and that they are, in fact, scoring the exercises 
according to the standards and guides that have been set 
forth. If any cases of discrepancy are noted, these can be 
corrected and the scoring adjusted so that the standards and 
guides are being followed as stated. 

The final aspect of the scoring of the exercises before 
their inclusion in the analysis program is the. scoring done by 
the optical scanning devices. The flexibility of the device 
allows for a complete range of consistency checks on the 
scoring as well as range checks on the numerical values 
assigned to the various parts of the exercises. This means 
there is an automatic screening or checking of each respojise 
ahd any exercise that has a response falling outside the 
ranges allowable, or that does not meet the consistency checks 
required by the standards , can be noted as a possible error to 
be corrected before its inclusion in the final process. 

The end result of all of the above requirements and checks 
is data that does meet the standards of the National Assessment 
program for atialysis purposes. The qualifications of the staff, 
the techniques that are set forth, and the checking of the end 
results all combine to provide the best possible scoring of the 
exercises, so that the data as reported is meaningful and 
constant. 
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CHAPTER 7 



PROCESSING- THE DATA 



The ECS (Education Commission of the States ) Data 
Processing Services Department (DPSD) was formed in May 1971 
to provide data processing support for the National Assessment 
of Educational Progress project and other ECS projects with 
primary emphasis on the National Assessment Project. Two 
cojnaputers are used to process the data — one located in Denver 
and one located at the Princeton University Computer Center — 
both connected by high speed telephone lines. 

Individual members of the DPSD staff are assigned the 
responsibility of Phase Coordinator fv^r each assessment year, 
with two or three other DPSD personnel as assistants. The 
Phase Coordinator is responsible for all data processing that 
takes place across all subject areas within an assessment year 
and is also a member of the NAEP subject area teams. In this 
way^ he becomes familiar with all phases of the National Assess- 
ment Project — objectives development^ exercise development, 
administration of the exercises, scoring specifications, and 
so on. 

Student File 

Approximately three months before the first MRC (Measurement 
Research Center) data tape is scheduled to arrive at the 
Princeton University Computer Center, DPSD personnel start 
designing and coding the NAEP student file. This file contains 
all information pertaining to each respondent. To ensure that 
the coding of the student file is correct, sample assessment 
packages are scored by MRC. This provides test data that is 
then sent to DPSD approximately one month before "live" data 
(the actual field administration results) is scheduled to 
arrive. The test data allows DPSD to find any program errors ' 
that might exist and make corrections before the live data is 
received. Magnetic data tapes arrive from MRC every six to 
eight weeks for each age level beginning with the exercise 
data for 13-year-olds followed by the 9-year-olds, in-schoo!. 
17-year-olds, out-of -school 17-year-olds, and adults. Data for 
each age level is added onto the NAEP student file as DPSD 
receives it. 

Work> Files 

The student file, which contains all the information 
collected for each respondent, is very large and therefore 
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cumbersome and costly to use in the many analyses required by 
NAEP. Therefore, smaller files, called work files, are 
created, each of which contains only the data required for the 
specific analysis to be performed.-. The programming . staff , 
upon receipt of specifications from the analysis staff, writes 
computer programs to perform the desired computations. Summary 
files are created to store the results of there computations. 
The results and appendices reported in the summary and theme 
volumes can then be obtained from the summary file without 
having to redo the desired computations each time. 

Appendix File 

After the summary files are complete, the appendix file is 
created. This file is used to print a final report with a 
short description of the exercise, the national percentage of 
success, and group differences from the national percentage of 
success for each exercise, as well as standard errors and other 
documentation. 

Final Reporting . • . . 

At this point, DPSD is ready to begin the final reporting 
data analysis phase. The Research and Analysis staff sends 
specifications to DPSD which computes reports in many formats* 
statistical tables, data-distribution plots, and any other 
format requested by NAEP. A data ■ base/data management system 
that IS publicly available and widely used called NIPS is used 
to process the NAEP data. It is designed to retrieve larae 
amounts of data quickly and efficiently. 
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CHAPTER 8 



DESCRIBING NATIONAL ASSESSMENT DATA 



Once an assessment has been completed, the responses 
scored, and the results analyzed, we become concerned with 
describing- thje results to educators and the concerned public 
in a manner that is understandable, meaningful, and useful* 
Since National Assessment data differ from typical test data, 
it is important that the reader have a clear understanding of 
our methods for describing data. National Assessment is not 
concerned with obtaining. individual scores as are testing 
programs. We are concerned with the percentage of individuals 
in the nation as a whole and in certain groups (see chapter 3) 
who possess various knowledges, understandings, skills, and 
attitudes and with comparing the percentages to each other. 

Percentage of Success 

An exercise which assesses a knowledge, an understanding, 
or a skill has a correct/incorrect criterion; i.e., the 
exercise has a definite correct response. Essay exercises in 
the Writing assessment are exceptions to this rule, because they 
are assigned overall quality scores. An exercise which 
assesses an attitude — in many cases — has a desirable/undesirable 
criterion; i.e., the exercise has no definite correct response, 
but certain attitudes are considered to be more desirable than 
others. In some instances, it is not practicable to call an 
attitude either desirable or undesirable. We refer to correct 
and desirable responses, collectively, as acceptable , and we 
deem essays acceptable or unacceptable on the basis of their 
overall quality scores. The percentage of success for an 
exercise is the percentage of participants who made an^ 
acceptable response to that exercise. It is the number of 
respondents in, a group who gave an acceptable response divided 
by the total number of respondents and multiplied by 100. 
Suppose, for example, there are 1500 respondents in a group; 
arid, of these, 650 give an acceptable response. The percentage 
of success for the group would be: 

# of acceptable , total # of x 100 = % of success 

responses " responses 

650 T 1500 X 100 = 43.3% 
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uh^i»"® express a percentage of success for the nation as a 

SU Vol iZrili °l If. L^f f "^^ Assessment groups (see cLpte? 
^^LUhJl example, a 64% of success for 9-year-olds in the ■ 
Northease region on a given exercise means that 64% of the 
9.year-old respondents in the Northeast region gave a^ accept- 
able response to that exercise. Responses to those atti?Sde 
■ n^f """"^x/^"^^ classified as desirable or ^ 

hSv!ng%tat lllil?a°eT' ^^^^^^ ^^^^ ^"^^-^^ 

Derived Scores 

xMany exercises have several parts, each of which can 
scored acceptable/unacceptable. ?or such an exerSlse, we 
often report the percentage of respondents who gave correct 
responses for none of the parts, for one part, f Sr tSS 
parts., and so on. In the second case, we repirt the 
gave correct responses for none of the partsT for one oart for- 
two parts, and so on. In the second casefw^ repo?? the " 
^w^'^fJ-^^^ °^ respondents who listed zero things" one thing 
two things, and so on-; " We call these percents derived scores. 

Comparisons among Groups 

for. .^"^ instances, we compare the percentage of success 

fs a whoJe'' TnAt^l^ Percentage of success for ?he nation 
An ffSSi • is called an effect. 

An effect is expressed as the percentage of succes s for a 
roTLli^lV''^ percentage of success lor the Slt^'on'Ss I whole. 

^ Northeast _ National- _ Northeast 
■s of Success % of Success ~ Effect ' 

^ positive effect means that a larger percent of r*:.Qr,onri^n+-„ 

so L'?bnSIon"2 acceptable responi^arexS"iIf Hir^fT'' 
so in the nation as a whole.. For example, if 74% of 

l.l7tTy°ltl o? g^yel^ol^firSe'L^^^'"^"^ ^^f^^^^' 
^^^^ 4- ui year-oias m the nation as a whole aave rin ' 

S be:' Northeast effect for 13-year-?lds , 

Northeast _ National _ Northeast 
% of Success % of Success " Effect 



" 68% = +6% 

= «v«„r, — ^ smaller percent i-cow^nu- 

a group gave an acceptable response to an exercise than 



ents ?n "^^arnn^ ;f';^L "'!f!lf J^,^^^ Sinaller percent of respond- 
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and-type of community, region of the country, sex, color, and 
level of parental education is to describe what differences 
(if any) actually exist in the real world between- members of 
various groups and the nation as a whole. 

When we look at the results for groups within one 
characteristic at *a time, however, we cannot say that a group 
effect is attributable solely to individuals being members of 
a group in question. A group — or the characteristic within 
which the group occurs — must not be construed as necessarily 
being the cause or even a cause of the effect associated with 
the group. 



\ 
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population of 25 marbles. There are 53,130 possible samples 
of five marbles that could be selected. If we select a sample 
of five marbles in such a way that each of the 53,130 possible 
samples has an equal probability of being selected, we can 
state that our sample is representative of the population. 
Some of the possible samples would have 80% (or four) blue 
marbles and 20% (or one) red marble — the exact same percentages 
as the population. Other samples, however, v/ould have various 
other percentages of blue and red marbles. Therefore, we 
cannot state with absalute certainty that the percentages of 
blue and red marbles in the population is exactly the. same as 
the percentages of blue and red marbles we obtain from a single 
selected sample. The differences between the percentages o.f 
^,biue_ and^ red™marbles obtained— f-rom-Tthe-^ample and the true 
percentages of blue and red marbles in the population is 
sampling error . 

This example with marbles shows how sampling error can 
occur, even when a sample is representative of the population 
from which it was selected. In the marble example, we . knew the 
population facts; but in the case of National Assessment per- 
centages of success and group effects, we do not know the 
population facts. • Concommitantiy , we do not know the extent or 
sampling error for any given sample percentage of success of 
group effect. It is for this reason, we cannot make exact 
statements about population facts on the basis of sample data. 

'We can compute a statistic called the standard error .for 
any sample percentage of success or group effect. A standard 
error is an estimate of the variation that would occur among 
the percentages of success or among the group effects for all 
potential samples that could be selected from the same 
population. We use the standard error and other statistical 
conventions to make statements about population percent^qes of 



ir-^- jj/j.v;s.ci4L.a a wixttj. liuri-tecnnicai aescription 

of the selection process; a more complete description of the 
process and its mathematical basis follows in chapter' 10. 

The primary purpose for developing a selection" procedure 
was to insure that, although exercises would be selected 
randomly, they would nontheless be representative of the total 
pool of exercises available. . With such a procedure we can be 
reasonably certain that a report provides coverage across • 
obDectives and across all population group differences to the 
extent that such coverage exists in the entire pool of 
exercises assessed. 



It is critical in a report that include 
the exercise assessed that we have exercises 
entire spectrum of data. For example, there 
for which males do much better than females, 
show no difference between males and females 
which the females do much better than the ma 
procedure enables us to achieve this kind of 
coverage for each group, i.e. males, females 



s only a portion of 
which represent the. 
should be exercises 
exercises which 

, and exercises for 

les. Our selection 
representative 

/ NE, SE, etc. 



After a computer randomly selects exercises for reporting 
we fill whatever gaps remain by looking for exercises which will 
provide us with an example of the type of data which is not in 
the set of exercises to be released. For example, if none of 
the exercises that were selected for release represent a large 
female advantage, one or more exercises are selected specifically 
because they show a large female advantage. This systematic 
selection proved necessary for the year 02 Reading report but 
not for Literature. ^ f ^ 
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will include the population percentage of success. The problem 
lies in the definition of reasonably; i.e., how confident 
should we be? The answer to this question resides in two risks 
accompanying the establishment of a confidence interval which ^ 
probably includes a population percentage of success. If we 
establish a confidence Interval based upon a degree of confi- 
dence approaching certainty, the size of the confidence interval 
could be so large as to be useless. On the other hand, if we ' 
establish a confidence interval based upon a low degree of 
.confidence, the size-of the confidence interval would tend to 
be small; but there would be a large probability that the 
interval is not wide enough to include the population percentage 
of success. We need to compromise these two risks so that, while 
we can be reasonably confident-^ that the interval includes the 
population percentage of success occurs, the interval is not so 
large that it tells us little about the value of the population 
percentage of success. The size of any given confidence inter- 
val depends upon the value of the standard error and the degree 
of confidence with which we wish to state that our interval ' 
includes the population percentage of success. 

This is best shown by considering three examples using 
hypothetical data. In all three examples, we use exactly the 
same sample percentage of success to illustrate that inferring 
a population percentage of success from a sample percentage of 
success is not a simple process. 



data: 



Example 12-1 . Let us assume we have the following sample 



Percentage of Success: 70% 
Standard Error: l% 



We wish to establish a confidence interval which we can, state 

with rnn f T /liayv7-»is i viv^l iit^^c =.^^ ^^u^ 



Using systematic selection of 10 to 15 percent of the' 
exercises, we are able to identify a set of exercises which are 
truly representative of the subject area being reported* In 
year 02, the selected exercises amounted to about 50% of all 
exercises. 
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CHAPTER la 



DETAILS OF SELECTING EXERCISES 
TO BE REPORTED 

In this chapter, we examine the details of the mathe- 
matical procedures used to select exercises to be reported. 
We give simple examples for each step of the selection 
procedure so that the rationale (see chapter 9) and the 
procedure itself can be more clearly understood and duplicated 
if desired. Since the selection process differed slightly for 
Reading and Literature, we describe the general process first 
and then those procedures specific to each subject area. 

Since the exercises not selected for release during the^ 
first assessment cycle are withheld to be released during tne 
second assessment cycle, those exercises designated for 
immediate release and those withheld for later releas? should 
be equivalent in two ways. First, both sets of exercises "inust . 
be equivalent in their coverage of objectives, themes, 
exercise formats and/or any other relevant characteristics. 
Second, botn sets of exercises must be statistically 
equivalent; i.e., they must have similar representation across 
the entire spectrum of percentages of success. This latter 
requirement prevents currently reporting, for example, that 
girls can read charts better than boys (on the basis of the 
released exercises) and then reporting five years hence that 
boys can read charts bett'^r than girls (on the oasis of the 
unreleased or withheld e^iercises) • The same consideration 
applies to all reportina categories. 

In order to insure the necessary equivalence National 
Assessment selects exercises in two steps. First, we group 
the exercises by their nonstatistical characteristics (objec- 
tive, theme, format, etc.)! Then within each of these 
groupings, we attempt to achieve statistical equivalence by 
developing an index which reflects group differences and can 
be used to form sets of similar exercises. 



These groupings vary from one subject area to another, 
depending upon the nature of the exercises in a given subject 
area. 
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The groups^ used to construct the selection index are: 
Type of Community (TOC) 

Extreme Inner City 
'Extreme Affluent Suburb 
Extreme Rural 

Other (all other STOC categories — see chapter 3) 

Color 

Black 
vJhite 

Sex 

Male 
Female 

Parental Education 

Wo High School 
Some High School 
Post High School 

Region ^ • 

Wortheast 
Southeast 
Central 
West 

National % 

Statistical Procedures 

P^values. We compute p-values for each of the groups 
listed above. A p-value is the proportion of respondents 
v/ithin a group who gave an acceptable response to an exercise 
that is, it is the number of respondents in a group who gave 
an acceptable response, divided by the total number of 



These groups are defined in chapter 6. 
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respondents in the group. 3 Suppose, for example, there are 
1500 respondents in a group; and, Id f" these, 650 gave an 
acceptable response. The p-value for the goup would be: 

# of acceptable Total # of ^ p--value 

responses responses 

650 1500 .433 

Arcsine - transformation . P-values aire easily inter- 
preted, but they possess unwieldy mathematical characteristics 
when they are used to construct an index. Therefore, the 
p-values were converted by the arcsine transformation: 

y = 20 arcsin p - 15.71 where: 

p = the p-value for some group; and 

y = the transformed p-value measured in radians 
with a possible range of -15.71 to +15.71. 

The arcsine transf ormation^f or the p-value computed in . 
the example above would be: 

y = 20 arcsin p - 15.71 

y = 20 arcsin .433 - 15.71 

y = 20 arcsin. (.658) - 15.71 

y = -1..3J/ 

The value of y is positive when the p-value is larger than .50, 
negative when the p-value is smaller than .50, and exactly 
zero for a p-value of .50. 

Effects . Once the original p-value have been trans- 
formed into y-values, we form effects by subtracting the 
National y value from the y-value effects for each of the 
groups used to construct the selection index (see p. 68). 



'A p-value multiplied by 100 gives the equivalent percentage 
of success (see chapter 8) . 



Exhibit 9-1 



Effects Used in the Selection Index 



Index Label 



Effects 



I 
S 



y (Extreme Inner City) - y (National) 
y (Extreme Affluent Suburb) - 

y (National) 
y (Extreme Rural) - y (National) 

(Other TOO - -y (National) 
y(Male ) - y (National) 
y(Female)*- y (National) 
y (White) - y (National) 
y (Black) - y (National) 
y(No High School) - y (National) 
y(Some Hign School) - y (National) 
y(Post High School) - y (National) 
y (Northeast) - y (National) 
y (Southeast) - y (National) 
y (Central) - y (National) 
y (West) - y (National) 
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W 

B 
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Suppose that on the same exercise for which we computed a 
hypothetical p-value and arcsine transformation, the National' 
p-value is .39. The corresponding y-value is -2.15. The 
effects, therefore would be: 

y (group) - y (National) = Effect 
-1.33 (-) -2.15 = +0.82 

The valuG of an effect is positive if the group p-valu'e is 
above th(5 National p-value and negative if the group p-value 
is below the National p-value. t 

The selection index . The selection index is a linear 
combination of effects for each exercise. We choose the 
effects to be used in the selection index (SI) on the basis 
of their variabilities and their correlations. If an 
effect has a limited range of values (Low variability) , all 
the p-values for a group are nearly identical for all exer- 
cises; therefore, we need not be concerned about the released 
and unreleased exercises being different with respect to that 
group. Similarly, if two effects are highly correlated/ it may 
suffice to consider only one of them since they provide redundant 
.information. Based upon- these criteria, the selection indices 
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for Reading and Literature differed somewhat. For Reading: 

SI = -I + S + W - B - NHS + PHS + NE - SE.4 
For Literature: 

SI = - I + s - B - NHS + PHS - SE. 
Selecti on of Reading Exercises 

based^I;non°S^!''iP''??''^"'.^°^ selection of exercises 

by their selection indices from those havina the la?apJ? 
positive values through zero to those haviSg ^he la^gelt 
negative values; for example, +2 +1 n -i ? mu^ 
then randomly selected one-?hi^d'o^ ^he'exeiclsis ?h?sTf " 

it has been selecIS fo^ reLafe b5 ?h^^ ^''J"^" inai«"n9 that 
grammed on the compute?f ^ P-^o^ea^-^e pro- 



«These contrasts are defined in Exhibit 9-1. 
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Exhibit 9-2 



Distribution of Selection Indices and One-Third 
Selection of Exercises 



Cluster 


Exercise # 


Selection Index 


Selected 


I 


NOl-05 
N03-11 
Wll-01 


11.36 
10.97 
9. 86 


* 


II 


NO- 02 

NOl-07 

N08-12 


6.33 
6.32 
5.98 


* ■ 


III 


N04-11 
Wll-02 
N02-11 


4.41 
3.92 
1.76 


* 


IV 


NO2-05 

NOll-06 

N03-14 


0./2 
0.19 
-0.16 


* 



Again consider the same 12 exercises we used in the 'random 
selection example above. Suppose that when we examined the 
Northeast contrasts we found that the four randomly selected 
exercises represented only the relatively large Northeast 
effects as shown in Exhibit 9-3. Exhibit 9-3 shows tiiat the 
four randomly selected exercises represent only the upper half 
of the total range of the Northeast abilities assessed by the 
12 exercises. In order to obtain a more accurate representa- 
tion of the entire spectrum of Northeast abilities, the set. 
of randomly selected exercises must be systematically augmented 
to include some exercises representing the poorer Northeast 
abilities, "^or example, by selecting exercises NOl-07 and 
N02-13 O£N03-"14, we have our desired 50% of the exercises 
selected and the entire Northeast range of abilities is 
represented. 

To the extent that exercises at different age levels were 
unique, the random-systematic selection procedure was executed 
separately for the age levels. Some exercises were -administered 
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at more than one age level. If such an exercise were selected 
for release at one age level, it would automatically be 
released at any other age level to which it was administerd. We 
are reporting all of a limited number of exercises concerned 
with skimming and scanning or reading rate. 



Exhibit 9-3 

Distribution of Northeast Contrasts and One-Third 
Selection of Exercises 



Exercise # 


NE Contrast 


Selected 


N03-11 


2.67 


* 


Nil- 01 


2.63 




N08-12 


2.55 


* 


N02-.05 


2.43 


* 


NOl-05 


2.36 




N04-11 


2.19 


* 


N02-02 . 


1.96 




Nll-02 


1.87 




NOl-07 - 


1.53 




Nll-06 - 


1.32 




N02-13 


1.09 




N03-14 . 


0.76 





Selection of Literature Exercises 

The, Literature exercises v^re grouped first by objective 
then by exercise format within objective.. Finally, within this 
structure, exercises, were iaped by the age level or the com-' 
bmation of age levels (when administered at more than one age 
Zu-l-^- th-y were administered. The exercises within 

this hierarchical grouping were^rdered by their average (mean) 
selection index for all ages from those 'having the largest 
positive values through zero to those having the largest nega- 
tive vaiues; for example +2, +1, 0,-1, -2. The computer then 
randomly selected one-half of the exercises in the pool by 
designating one exercise from every cluster of two exercises in 
each Hierarchical grouping. Suppose, for example, there are 12 
exercises m such a grouping and. we want to select one-half of 

tnZ^c f^rf I^i^^S^K^^^u" exercises by average selection 

index (SI) would be shown in Exhibit 9-4. Note that the 
average selection index descends in value and that one exercise 
m eacn cluster of two has been starred indicating that it has 
oeen selected for release by the random procedure pJog^a^med on 
on the computer. No systematic selection was doL^for SuIratSre. 
All exercises wnich involved direct interview and/or tape 
recorded responses are being reported. c^na/or rape 



Exhibit 9-4 



Distribution of Average Selection Indices and One-half 

Selection of Exercises 

Cluster Exercise # sT Selected 

I . NOl-05 11.36 * 

N03-11 10.97 



II Nll-01 9.68 

^N02-02 —6.-33 



III NOl-07 6.32 

N08-12 5.98 



IV 


N04-11 


4.41 


* 




.Nll-02 


3.92 




V 


N02-11 


1.76 


* 




N02-05 


0.72 




VI ■ - 


Nii^oe 


0.19 






N03-14 


-0 16 


* 



CHAPTER 11 



OVERVIEW OF INFERRING POPULATION FACTS FROM 
SAMPLE DATA 



Populations and Samples 

A population includes ail the individuals in some defined 
group. Examples of some National Assessment populations are: 
all 9-year-olds living in the Central region of the nation or 
all IV-year-olds living in the Extreme Inner City. In our 

r^tn:tsr7*n?7e"^w±^sh-^o"~^ 

skills, and attitudes of entire populations. It : ' usually 
not practicable, however, to obtain data from entire populations 
and for National Assessment to do so would be impossible. 
Therefore, we obtain dat^ from a portion of the total popula- 
tion called a sample . ' If we select a sample with care in 
accordance with ^certain rules, we can say that the sample is 
representative of the population from which it was selected and, 
likewise, that the data obtained from th'^ sample are repre- 
sentative of the data we would obtain from the entire population. 
National Assessment samples have been selected in such a way that 
they are representative of the populations from whxch they were 
selected. 1 

From Sample Data to Population- Facts , 

The advantage gained by obtaining data fiom samples rather 
than from populations is somewhat qualified by a loss, of 
precision in the descriptions we can give of populations on the 
basis of that data. Within the limits of error due to measure- 
ment, the data we obtain from a sample precisely describes that 
sample. Even when we have a sample which is representative of 
the population from which it was selected, we cannot state with 
absolute certainty that the data we obtain from it is . exactly . 
true for tne raspective population. This is because there are 
many .potential samples which could have been selected from the 
same population — all selected with equal care and in accordance 
with such rules that each one would be representative of that 



Chapter 4 gives a detailed description of the National 
Assessment sampling procedures. 

■ ■ - 'A 
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population. Lven with all this care, one would not expect to' 
obtain exactly the same percentages of success or group effects 
from all these samples. The variation that would occur among 
the percentages of success or- among the group effects obtained 
from all the potential samples which could have been selected 
from the same population is called sampling error . 

. VJe can compute a statistic called the standard error for 
any percentage of success or group effect. A standard error 
is an estimate of the variation that would occur among the 
percentages of success* or among the group effects for all poten- 
tial samples that could have been selected from the same 
population — the larger -izhe standard error the greater the 
varia^ionr The larger the standard error^for a^percentag'^^ ^ 
success or a group effect, the less precise is the statement we 
can make, about the population percentage of success or group 
effect. 



The details of the development of the limits discussed 
above and their rationale are given in chapter 12. 
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CHAPTER 12 



DETAILS OF INFERRING 
POPULATION FACTS FROM SAMPLE" DATA 



In chapter 11, we defined populations and samples and 
explained why it is necessary for National Assessment to 
obtain data from samples rather than entire populations. We 
also noted that there is a loss of precision when we describe 
population facts on the basis of data obtained from samples. 
This chapter presents the details of and the rationales for the 
statistical procedures which allow us to infer population fa cts 
from sample data and to describe the degree of imprecision 
involved. 

Sampling Error ^ 

VJhenever we infer population facts from sample data, we 
must bear. in mind that a very large number of samples could 
' be selected from, the same population — all selected with equal 
care and in" accordance with such rules that each one would-be 
representative of the population from which it was selected. 
Even with all this care, we would not expect to obtain exactly 
the same percentages of success or exactly the same group 
effects from all these potential samples. The variation that 
would occur among the percentages of success or among the 
group effects for all these potential samples is called 
sampling error . The concept of sampling error is important 
when we irif er population percentages of success and group 
effects from sample percentages of success and group effects. 
Because of sampling error, we cannot state with absolute 
certainty that the value of the population percentage of 
success or group effect is exactly the same as the value of the 
obtained sample percentage of success or group e'rfect. Most of 
the percentages of success and group effects for all potential 
samples would be quite close to the population percent of 
success and group effect, but a few would differ by a large 
degree. . \ 

In order to see more clearly just how this works ,^ let us 
consider a bag containing 25 marbles — 20 blue marbles and five 
red marbles. The 25 marbles are the population and the 
population facts are: there are 80% (or 20) blue marbles and 
20% (or five) red marbles. Suppose, However, that we do not 
know the population facts and want to estimate them on the basis 
of data obtained from a sample of five marbles selected from the 
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Therefore, we can state with 95% confidence that the interval , 
68'o--72% includes the population percentage of success •3 

Example 12-2 > Let us assume we have the following sample 

data: 

Percentage of Success: 70% 
Standard Error: 3% 

Again, we wish to establish a confidence interval which we can 
be 95% certain includes the population percentage of success. 
We- use the same statistical convention to establish the confi- 
dence interval as we used in example 12-1. Our confidence 
interval would be between 70% --(1.96 x 3%) and 70% +(1.96 x 3%) 
or between 64% and 76%. Therefore, we can state with 95% 
confidence that the interval 64% — 76% includes the population 
percentage of success. 

Example 12-3 . Let us assume we have the following sample 

data: 

Percentage of Success: 70% 

Standard Error: .1^ 

These are the same data we used for Example 1*2-1; but for 
Example 12-3, we wish to establish a confidence interval which 
we can state with 99.8% confidence includes the population 
percentage of success. We use the statistical convention that 
when we have a large number of observations, 99.8% of those 
observations occur within 3.09 standard error units on either 
side of the mean. Our confidence interval would be between 
70% -(3.09 X 1%) and 70% +(3.09 x 1%) or between 67% and 73%. ' 
Therefore, we can svate with 99.8% confidence that the interval 
67% — 73% includes the population percentage of success. 

Summary . Because of rampling error / we cannot infer that 
a population percentage of success is exactly the i^ame as the 



If we select 100 samples and establish confidence intervals ' 
around the 100 obtained percentages of success in the manner 
described, on the average, 95 of them would include the - 
population percentage of success. 
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1. 



2. 



percentage of success obtained from the sample. We can 

es^ab??«h"r.on?.f ^"^^^^^'^'^^f statistical conventions to 
establish a confidence interval which we can be reasonably 
certain includes the population percentage of success. On the 

tSJiiJf nf '^^^^ examples, we can geneLte two rulJ^ regarding 
the size of confidence intervals. ^"xny 

When a sample percentage of success has a large 

standard error, the size of the confidence 

interval must be larger than when the sample per- 
centage of success has a small standard error in 
order to state with an equal degree of confidence 
that the interval includes the population 
per centage of succ ess. Com paring Exanipif> „12^^ 
with Example 12-2, note that in order to state 
witn 95% confidence- that the population percentage 
of success occurs within the interval we need an 
interval three times as large for Example 12-2 
as for Example 12-1. 

For any given standard error of a percentage of 
success, the size of the confidence interval 
must be larger when we wish to state with a 
hi^h degree of confidence than when we wish to 
state with a lower degree of confidence that the 
population percentage of success occurs within 
the interval. In comparing Example 12-1 with 
Example 12-3, note that both examples have the' 
same standard error, but the 99.8% confidence 
interval (Example 12-3) is one and one half 
times as large as the 95% confidence interval. 

A confidence interval should not be regarded as a mystical 

ITeT/^tTifT^ttu'^^^l by sta^:isticians; wl encounter ?hem In 
everyday life— altnough not m the strictly formal sense we ' 
silLZtt ^^f-^h^" not possible to make prlcisS 

me?cSS; ^^^"^Pl^' that yoa order an article of 

d^??wf^- the salesiT.an tells you that the article will 
has IsJISf^hiS ^0 days give or take two days. The salesman 

^ f confidence mterval-the give or take two . 
days-around tne average time it has taken that article'to be 
d^ii'^So P^t? e"'^. within which he is reasonably confi- ' 

?n?^r.v??="^^''^^ ^t-^^f delivere- this time. The confidence 
intervals we establish for percentages of success are much the 
same • 
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From Sample Effects to Population Effects 

If a group population were compared to the population for 
the nation as a whole / we would never expect to obtain an 
effect exactly equal to zero; i.e., some difference^ however 
small, would exist. Many population effects, however, would be 
so small that they can be called equal to zero for practical 
consideration. 

When we infer a population effect from a sample effect,, we 
are concerned, first of all, with three considerations. 

1. Is a sample effect large enough that we can be 

_ reas onably certain that the population effect is 
not equal to zero; i.e., does a real difference 
exist between the group and the nation as a whole? 

2. Is the sample effect small enough that we can be 
reasonably certain that the population effect is, 
for practical consideration, equal to zero; i.e. , 
does no difference of practical magnitude exist 
between the group and the nation as a whole? 

3. Is the sample effect . not large -enough that we can 
be reasonably certain that the population effect 
is not equal to zero and not small enough that ■ 
we can be reasonably certain, that the population 
effect is, for practical consideration, equal 

to zero; i.e., can we make no reliable statement 
as to whether a real difference exists or whether 
no practical difference exists between a group and 
the nation as a whole? 

When we infor a population effect from a sample effect, 
we' are concerned, secondly, with the risk of making two types 
of error which are related to the three considerations stated 
above. The first type of error is calling a population effect 
not equal to zero when, in fact, the population effect is, for 
practical consideration, equal to zero. The second type of 
error is calling a population effect, for practical considera- 
tion, equal to zero when, in fact the population effect is not 
equal to zero. We can never eliminate the risk of making both 
types of error— -or even one type of error- -completely. 
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CHAPTER 13 



THEMES 



Beginning with the year 02 assessment (Reading and 
Literature), we are reporting the results for, each subject area 
in a series of theme reports.! Each report presents the results 
a group of exercises which share a common theme or unifying idea. 
For example, one of the Literature themes is "Understanding the 
language of literature." A report volume presents the results 
for a variety of different exercises which were designed to 
assess an individual's knowledge of the way language operates 
in poetry and prose. In Reading, one of the. themes is "Read-^~ 
for main ideas and organization of passages." This theme 
embodies exercises which require individuals to read' a passage 
and detect its central thought or topic or to detect the mode 
in which the material is organized. 

The conciept of themes arose from the desire of National 
Assessment to report its results in a manner meaningful to 
educators and other concerned persons. If we divide the entire 
pool of exercises for a subject area into sets of exercises 
which have a common theme, educators should be able to ani^wer 
such questions as the following: 

1. Do various groups perform differentially in a 
given situation? 

2. How. does a given group perform in one situation 
as opposed to other situations? 

3. Does a group perform better on certain aspects 
of a given situation than on oth.r aspects of 
the same situation, and are thes.^ aspects 
identifiable .2 



^See Foreword, the section — Reporting Format. 
2 

*The themes for Reading and Literature are outlined and dis- 
cussed briefly in the Reading Summary and Literature Summary , 
respectively* Each theme is discussed in detail in the 
respective theme volumes. 
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The themes for both Reading and Literature were developed 
post hoc; that is, chey are based on only those- exercises 
which were actually administered. These themes do not exhaust 
all possible contents in the respective subject areas since 
the exercises which were developed for these subject areas 
were not, themselves, exhaustive. 

In each subject area, the pool of administered exercises 
was examined critically by several National Assessment staff 
members who categorized the exercises into themes. The popl 
of exercises was simultaneously sent to a consultant (a 
scholar in each respective subject area) , who also examined 
the exercises and- categorized them into themes. "When both 
the staff members and the consultant had arrived— independently 
.::atLreasonabAe_sets of themes, the staff members and the 



consultant meh to resolve differences and to finalize a 
consensus" set of themes. 

Once viable themes have been developed for a subject area 
they serve two somewhat related functions. . First, each of 
them serves as the content of a report volume. Second and more 
importantr-theme reports allow educators and other cSncSned 
persons to examine in a single volume the results of a' 
meaningful set of exercises. Each theme volume gives results 
for the nation as a whole and the various groups defined in 
chapter 3, The reader of these reports can examine behaviors 
of groups for the theme in general (as expressed by the 
median-}) or for specific exercises. 



3 

The median is the mid-point of the range of- percentages of 
success for all the exercises within a theme. A median can 
be computed for the nation as a whole and for each group. 
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CHAPTER 14 



THE MERITS AND WEAKNESSES OF ADJUSTI4ENT 
(INCLUDING BALANCING) 

The educational administrator wants to make comparisons 
between groups to find out who is learning more and who less 
in hopes of being able to improve performance in the laggina 

racllls to^'tTataJ'lnT^'^ '''^ ^° '""^^^^ find oSfwLt 
s?rena?h^n iuT^^ and how much changes in these factors would 
strengthen the educational achievement of the students affected 
For example, when we find that boys know less about the 
reproductive system of both sexes than do girls, this raises at 
^S^vi^^hiS^'ff strengthening the educatioi of boys 
l^Ti^^tl^ ^^^^""^ ^° subdivide the country into finer 

"'^ke comparisons between subgroups 
that have "everything alike" except the variable being studied. 

n ^ P ^^^f"" words, we search for causes of the differences 
nrfb^^Jif'^^i^' we cannot have "everything alike" in social 
problems and rarely in physical problems either, and so we are 
not actually able to carry out the precise programs; But 

?h=r \ ^^^^^"^ ^^^^ we may carry out 

J the program that seems feasible. We subdivide bv 
groSps ^^^"^^^^^ ^^^^ comparisons in performance^ong"^ 

variah?p<,^?ho^»,^^J^ happens" is that as we introduce several 
For ex^nl^ 1? "^^J °^ subdivisions grows like a product. 
For example, if we have 5 variables with 2, 3, 5, 7, and 4 
categories respectively, we have 2x3x5x7 x 4 = 840 - 
subgroups, and a sample of 8400 people would give an averaae of 
only 10 per subgroup. Naturally many subgroups woSd bS empt? 
possme S wil/stil! be'^hard, if^noHm^ 

too sparse? comparisons among subgroups, for some will be 

at facLrs'oL'J? Tt^^ '"^^^ ^^^^^^ ^^^^^ ^-^-^ 

more IZT't'^U^^^^^^^'' t"" Extreme Affluent Suburb tend, 

educated oa^ii??'' i"" '""^^ ^i^^' t° have better 

H???^? !* ^eca^se of this lack of balance, part of 

griwfng oS^of thf^?.^^^'^ "^^^ consiLred as 

growing out of the difference in parental education. 

these^ext?eSf?Jni.^°/^^' "^^^^ ^^'^ difference between 

of oLIntar^n^K- °^ •^^'^"^nity have been if the distribution 
of parental education, sex, color and region had been the same 
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for both types of community referred to above?" Were it 
possible to rearrange the world to equate these distributions 
for each type of community, the effects upon our nation and its 
schools would be profound. Such rearrangement is not possible. 
It is iisually appropriate to think , of the balanced results 
presented in a later special report as reflecting the dif f -rences 
we would see in the absence of masquerading by the other lour 
factors. We can be reasonably sure the balanced results do a 
much better job than the unadjusted results of reflecting such 
differences. 

Still another question concerns the combination of factors. 
The performance of a given group may be found to differ, 
depending upon subgroupings on other variables. Thus, the 
effect associated with Extreme Affluent Suburbs may be different 
in the Northeast and the Southeast.' Or the effect associated 
with sex may be somewhat different for Blacks and VJhites. 
Such interactive differences may be of importance; balancing 
does not adjust for them. 

It is natural to ask whether this or any such method of 
analysis can help us. To some extent they can aid, to some 
extent not. We cannot make up for cases we don't have but we 
may be cible to supply approximate analyses that will come near 
to answering such a question as what is the effect of region of 
the country on performance when you .control for size-and-type 
of community and several other variables. If the effect of 
region is . substantially reduced by the analytical adjustment, 
we may be inclined to think that region is not in itself the 
cause of the raw differences as much as the other variables. 
One role of adjustment then is to help us make approximate 
comparisons and summaries that we cannot make by directly 
subdividing all the variables. • i^'?- 



Elsewhere (see Foreword and chapter 8) we hav^'^many 
cautionary remarks about the dangers of misinterpreting the 
causative powers of given background variables, for they may 
be poorly measured and they may not mean what they say. For 
an example from the field of warfare, in World War II the more 
fighter opposition bombers had, the closer to the target were 
the bombs. Why? Fighters didn't come up when the weather 
obscured the target. ^ Such proxy variables, especially when 
their correct interpretation may be the absolute reverse of 
their obvious effect puts us in grave danger of making 
mistakes. We do not go further into that here. 
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A second ar(^umeht favors ad ju^^^ 'It argues that we 

:imust^^^a^ the 
adjiastment wiir r^ so that we show the potential 

vSf ^he d \ }^ :v ; 



^ " "S^ - ^^^^ " ^ ; - -85 



^ ' : . ' - ' \ \ - ; : ' r 

Nothing but -experimentation^ if that^ can serve to demon- T^' 

strate what the actual effect of changes will be. We are, ?r 

however, trying tP get hints and insights from the data we have. ' I 

Furthermore, if someone does have a causative model involving |\ 

the variables National Assessment measures, he does have a J 
chance to check it against these results. 

We see then that the purpose of analysis and adjustment is ; 

to help the data reveal information that they cannot give in i 

their raw form. Aside from the dangers of misinterpretation, f 

we have the pplitical arguments for and against adj* fcment. i 

First, against: if adjustment for background variab^as seems | 

to reduce the differences between a group of the population and 4 

^ the national average, it has been argued that this tends to v | 

minimize the disadvantage of the group and, it is further I 

argued, that adjustmelit should not be made. The direction of _ | 

the effect of an adjustment is not necessarily one-way; | 

adjustments can increased d^^ as well as decrease them. - |- 
Those arguing ^ . . |^ 

^presumably argue for it in the qase of increased discrepancies. 



Clearly the making the first and second argument 

^want the same thingf , to __impro the position of the. dis- _ _ 

a^^^ this is a national goal. Steps - ^ t 

toward achieving such goals d^^ oh searching for pauses _ 1 

amJ; methods of imprpyement, on finding weak spots in a system . |i 
aiid sp^o We should, theref pre, look at our ^d^ _^ 

;way we can for hints about how the system works and how to _| 

limprove it. Analysis and adjustment are tools for doing this. | 
;The qu<^stion is not whether tp adjust or not, but, "What are 
the useful ^ys?^* and, "What' do the variables mean?" , \ ' p 

:>"What further variables do we need to measure?", and "How shall |; 
we interpret - : / : - / _ \ : > > 



